# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7b2f8198-0513-4428-8bdb-21519bcfcb47
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7b2f8198-0513-4428-8bdb-21519bcfcb47\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e44faccf-cec5-4da6-a4ac-b4e0a633efee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4b2f68ccd663b927abd29ec9b531301ca0f06c1b5b18a2d1502343ff08cb7762\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4244b4d2-21f2-4754-a02c-441faaab7027\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f630e431d1fdbd9240d3ffeee36ab689018fc55224cd7525fb24a044a942153a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"61c3ab2a-513b-48f6-b3db-3f6b4737ab2f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f3759b436ce7e9a0576462292acf4dece303bc24ae4a08e06a544a7e6645a48\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2792, \"end_char_idx\": 5943, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "doc_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "ref_doc_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "chroma:document": "## Implementation differences with SwitchTransformers\n\nThe biggest difference is the way the tokens are routed. NLLB-MoE uses a `top-2-gate` which means that for each input, only the top two experts are selected based on the \nhighest predicted probabilities from the gating network, and the remaining experts are ignored. In `SwitchTransformers`, only the top-1 probabilities are computed, \nwhich means that tokens have less probability of being forwarded. Moreover, if a token is not routed to any expert, `SwitchTransformers` still adds its unmodified hidden \nstates (kind of like a residual connection) while they are masked in `NLLB`'s top-2 routing mechanism. \n\n## Generating with NLLB-MoE\n\nThe available checkpoints require around 350GB of storage. Make sure to use `accelerate` if you do not have enough RAM on your machine.\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-moe-54b\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-moe-54b\")\n\n>>> article = \"Previously, Ring's CEO, Jamie Siminoff, remarked the company started when his doorbell wasn't audible from his shop in his garage.\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"fra_Latn\"], max_length=50\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"Auparavant, le PDG de Ring, Jamie Siminoff, a fait remarquer que la soci\u00e9t\u00e9 avait commenc\u00e9 lorsque sa sonnette n'\u00e9tait pas audible depuis son magasin dans son garage.\"\n```\n\n### Generating from any other language than English\n\nEnglish (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.\n\nSee example below for a translation from romanian to german:\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-moe-54b\", src_lang=\"ron_Latn\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-moe-54b\")\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"deu_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 61c3ab2a-513b-48f6-b3db-3f6b4737ab2f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"61c3ab2a-513b-48f6-b3db-3f6b4737ab2f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e44faccf-cec5-4da6-a4ac-b4e0a633efee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4b2f68ccd663b927abd29ec9b531301ca0f06c1b5b18a2d1502343ff08cb7762\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b2f8198-0513-4428-8bdb-21519bcfcb47\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"957d7f7d470b91fb776e88e8b678995fcd8463c15d45227c3acfdd4108ca7aab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4427, \"end_char_idx\": 6292, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "doc_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "ref_doc_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "chroma:document": ">>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"fra_Latn\"], max_length=50\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"Auparavant, le PDG de Ring, Jamie Siminoff, a fait remarquer que la soci\u00e9t\u00e9 avait commenc\u00e9 lorsque sa sonnette n'\u00e9tait pas audible depuis son magasin dans son garage.\"\n```\n\n### Generating from any other language than English\n\nEnglish (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.\n\nSee example below for a translation from romanian to german:\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-moe-54b\", src_lang=\"ron_Latn\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-moe-54b\")\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"deu_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n\n## NllbMoeConfig\n\n[[autodoc]] NllbMoeConfig\n\n## NllbMoeTop2Router\n\n[[autodoc]] NllbMoeTop2Router\n    - route_tokens\n    - forward\n\n## NllbMoeSparseMLP\n\n[[autodoc]] NllbMoeSparseMLP\n    - forward\n\n## NllbMoeModel\n\n[[autodoc]] NllbMoeModel\n    - forward\n\n## NllbMoeForConditionalGeneration\n\n[[autodoc]] NllbMoeForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d584921e-ed67-4381-bd25-1341de316a50
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d584921e-ed67-4381-bd25-1341de316a50\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e3fd24d-d149-4851-b64c-3be8e0a7de21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af4d63280a9c5ee224b1d6d527b5815b86fe2a71f11285d908624fb97903a6c0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7315d703-a61c-45c2-bef0-d0c89a1f7d49\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e1e69a561b5a68cd927ec376477ee5d56399eefd95e3659b365cd237db801918\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2702, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e3fd24d-d149-4851-b64c-3be8e0a7de21", "doc_id": "4e3fd24d-d149-4851-b64c-3be8e0a7de21", "ref_doc_id": "4e3fd24d-d149-4851-b64c-3be8e0a7de21", "chroma:document": "# GPTSAN-japanese\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe GPTSAN-japanese model was released in the repository by Toshiyuki Sakamoto (tanreinama).\n\nGPTSAN is a Japanese language model using Switch Transformer. It has the same structure as the model introduced as Prefix LM\nin the T5 paper, and support both Text Generation and Masked Language Modeling tasks. These basic tasks similarly can\nfine-tune for translation or summarization.\n\n### Usage example\n\nThe `generate()` method can be used to generate text using GPTSAN-Japanese model.\n\n```python\n>>> from transformers import AutoModel, AutoTokenizer\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Tanrei/GPTSAN-japanese\")\n>>> model = AutoModel.from_pretrained(\"Tanrei/GPTSAN-japanese\").cuda()\n>>> x_tok = tokenizer(\"\u306f\u3001\", prefix_text=\"\u7e54\u7530\u4fe1\u9577\", return_tensors=\"pt\")\n>>> torch.manual_seed(0)\n>>> gen_tok = model.generate(x_tok.input_ids.cuda(), token_type_ids=x_tok.token_type_ids.cuda(), max_new_tokens=20)\n>>> tokenizer.decode(gen_tok[0])\n'\u7e54\u7530\u4fe1\u9577\u306f\u30012004\u5e74\u306b\u300e\u6226\u56fdBASARA\u300f\u306e\u305f\u3081\u306b\u3001\u8c4a\u81e3\u79c0\u5409'\n```\n\n## GPTSAN Features\n\nGPTSAN has some unique features. It has a model structure of Prefix-LM. It works as a shifted Masked Language Model for Prefix Input tokens. Un-prefixed inputs behave like normal generative models.\nThe Spout vector is a GPTSAN specific input. Spout is pre-trained with random inputs, but you can specify a class of text or an arbitrary vector during fine-tuning. This allows you to indicate the tendency of the generated text.\nGPTSAN has a sparse Feed Forward based on Switch-Transformer. You can also add other layers and train them partially. See the original GPTSAN repository for details.\n\n### Prefix-LM Model\n\nGPTSAN has the structure of the model named Prefix-LM in the `T5` paper. (The original GPTSAN repository calls it `hybrid`)\nIn GPTSAN, the `Prefix` part of Prefix-LM, that is, the input position that can be referenced by both tokens, can be specified with any length.\nArbitrary lengths can also be specified differently for each batch.\nThis length applies to the text entered in `prefix_text` for the tokenizer.\nThe tokenizer returns the mask of the `Prefix` part of Prefix-LM as `token_type_ids`.\nThe model treats the part where `token_type_ids` is 1 as a `Prefix` part, that is, the input can refer to both tokens before and after.\n\n## Usage tips\n\nSpecifying the Prefix part is done with a mask passed to self-attention."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7315d703-a61c-45c2-bef0-d0c89a1f7d49
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7315d703-a61c-45c2-bef0-d0c89a1f7d49\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e3fd24d-d149-4851-b64c-3be8e0a7de21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af4d63280a9c5ee224b1d6d527b5815b86fe2a71f11285d908624fb97903a6c0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d584921e-ed67-4381-bd25-1341de316a50\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"690b2b3d16c3476b6ba34fdbc99ec28b45fd876bec2d9697a07b67d82041be49\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0ca55166-9c19-4f25-9b1a-a06a2064b0e6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"74a89b65a6c9a6dcf31a94c0d7d0c90b94e4a22402997d946631154377c4088c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1398, \"end_char_idx\": 3695, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e3fd24d-d149-4851-b64c-3be8e0a7de21", "doc_id": "4e3fd24d-d149-4851-b64c-3be8e0a7de21", "ref_doc_id": "4e3fd24d-d149-4851-b64c-3be8e0a7de21", "chroma:document": "It has a model structure of Prefix-LM. It works as a shifted Masked Language Model for Prefix Input tokens. Un-prefixed inputs behave like normal generative models.\nThe Spout vector is a GPTSAN specific input. Spout is pre-trained with random inputs, but you can specify a class of text or an arbitrary vector during fine-tuning. This allows you to indicate the tendency of the generated text.\nGPTSAN has a sparse Feed Forward based on Switch-Transformer. You can also add other layers and train them partially. See the original GPTSAN repository for details.\n\n### Prefix-LM Model\n\nGPTSAN has the structure of the model named Prefix-LM in the `T5` paper. (The original GPTSAN repository calls it `hybrid`)\nIn GPTSAN, the `Prefix` part of Prefix-LM, that is, the input position that can be referenced by both tokens, can be specified with any length.\nArbitrary lengths can also be specified differently for each batch.\nThis length applies to the text entered in `prefix_text` for the tokenizer.\nThe tokenizer returns the mask of the `Prefix` part of Prefix-LM as `token_type_ids`.\nThe model treats the part where `token_type_ids` is 1 as a `Prefix` part, that is, the input can refer to both tokens before and after.\n\n## Usage tips\n\nSpecifying the Prefix part is done with a mask passed to self-attention.\nWhen token_type_ids=None or all zero, it is equivalent to regular causal mask\n\nfor example:\n\n>>> x_token = tokenizer(\"\uff71\uff72\uff73\uff74\")\ninput_ids:      | SOT | SEG | \uff71 | \uff72 | \uff73 | \uff74 |\ntoken_type_ids: | 1   | 0   | 0 | 0 | 0 | 0 |\nprefix_lm_mask:\nSOT | 1 0 0 0 0 0 |\nSEG | 1 1 0 0 0 0 |\n\uff71   | 1 1 1 0 0 0 |\n\uff72   | 1 1 1 1 0 0 |\n\uff73   | 1 1 1 1 1 0 |\n\uff74   | 1 1 1 1 1 1 |\n\n>>> x_token = tokenizer(\"\", prefix_text=\"\uff71\uff72\uff73\uff74\")\ninput_ids:      | SOT | \uff71 | \uff72 | \uff73 | \uff74 | SEG |\ntoken_type_ids: | 1   | 1 | 1 | 1 | 1 | 0  |\nprefix_lm_mask:\nSOT | 1 1 1 1 1 0 |\n\uff71   | 1 1 1 1 1 0 |\n\uff72   | 1 1 1 1 1 0 |\n\uff73   | 1 1 1 1 1 0 |\n\uff74   | 1 1 1 1 1 0 |\nSEG | 1 1 1 1 1 1 |\n\n>>> x_token = tokenizer(\"\uff73\uff74\", prefix_text=\"\uff71\uff72\")\ninput_ids:      | SOT | \uff71 | \uff72 | SEG | \uff73 | \uff74 |\ntoken_type_ids: | 1   | 1 | 1 | 0   | 0 | 0 |\nprefix_lm_mask:\nSOT | 1 1 1 0 0 0 |\n\uff71   | 1 1 1 0 0 0 |\n\uff72   | 1 1 1 0 0 0 |\nSEG | 1 1 1 1 0 0 |\n\uff73   | 1 1 1 1 1 0 |\n\uff74   | 1 1 1 1 1 1 |\n\n### Spout Vector\n\nA Spout Vector is a special vector for controlling text generation."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0ca55166-9c19-4f25-9b1a-a06a2064b0e6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0ca55166-9c19-4f25-9b1a-a06a2064b0e6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e3fd24d-d149-4851-b64c-3be8e0a7de21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af4d63280a9c5ee224b1d6d527b5815b86fe2a71f11285d908624fb97903a6c0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7315d703-a61c-45c2-bef0-d0c89a1f7d49\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6dd5cb46b98bcc69986a45bafe4824e605918bb7490a26eec29410037b951b66\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3696, \"end_char_idx\": 4437, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e3fd24d-d149-4851-b64c-3be8e0a7de21", "doc_id": "4e3fd24d-d149-4851-b64c-3be8e0a7de21", "ref_doc_id": "4e3fd24d-d149-4851-b64c-3be8e0a7de21", "chroma:document": "This vector is treated as the first embedding in self-attention to bring extraneous attention to the generated tokens.\nIn the pre-trained model published from `Tanrei/GPTSAN-japanese`, the Spout Vector is a 128-dimensional vector that passes through 8 fully connected layers in the model and is projected into the space acting as external attention.\nThe Spout Vector projected by the fully connected layer is split to be passed to all self-attentions.\n\n## GPTSanJapaneseConfig\n\n[[autodoc]] GPTSanJapaneseConfig\n\n## GPTSanJapaneseTokenizer\n\n[[autodoc]] GPTSanJapaneseTokenizer\n\n## GPTSanJapaneseModel\n\n[[autodoc]] GPTSanJapaneseModel\n\n## GPTSanJapaneseForConditionalGeneration\n\n[[autodoc]] GPTSanJapaneseForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9c9a3cd5-259b-4c38-904a-7d0bb9b713a6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nat", "title": "Neighborhood Attention Transformer", "tokens": 1124, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9c9a3cd5-259b-4c38-904a-7d0bb9b713a6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c6fbfa21-2511-49eb-8aae-62456d8ddc5a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40d20cbbd35f13e4041026c403b647a26dd23e3c4d8fde2abcbeb775a9806eee\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"71118917-4b7b-4225-980c-585b7b23b89f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d30e68a883caf4322ea4140dd9e352882be2759bc9b73a03abb8af50571ea3c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3159, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c6fbfa21-2511-49eb-8aae-62456d8ddc5a", "doc_id": "c6fbfa21-2511-49eb-8aae-62456d8ddc5a", "ref_doc_id": "c6fbfa21-2511-49eb-8aae-62456d8ddc5a", "chroma:document": "# Neighborhood Attention Transformer\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nNAT was proposed in [Neighborhood Attention Transformer](https://arxiv.org/abs/2204.07143)\nby Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi.\n\nIt is a hierarchical vision transformer based on Neighborhood Attention, a sliding-window self attention pattern.\n\nThe abstract from the paper is the following:\n\n*We present Neighborhood Attention (NA), the first efficient and scalable sliding-window attention mechanism for vision.\nNA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels, and therefore enjoys a\nlinear time and space complexity compared to the quadratic complexity of SA. The sliding-window pattern allows NA's\nreceptive field to grow without needing extra pixel shifts, and preserves translational equivariance, unlike\nSwin Transformer's Window Self Attention (WSA). We develop NATTEN (Neighborhood Attention Extension), a Python package\nwith efficient C++ and CUDA kernels, which allows NA to run up to 40% faster than Swin's WSA while using up to 25% less\nmemory. We further present Neighborhood Attention Transformer (NAT), a new hierarchical transformer design based on NA\nthat boosts image classification and downstream vision performance. Experimental results on NAT are competitive;\nNAT-Tiny reaches 83.2% top-1 accuracy on ImageNet, 51.4% mAP on MS-COCO and 48.4% mIoU on ADE20K, which is 1.9%\nImageNet accuracy, 1.0% COCO mAP, and 2.6% ADE20K mIoU improvement over a Swin model with similar size. *\n\n<img\nsrc=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/neighborhood-attention-pattern.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Neighborhood Attention compared to other attention patterns.\nTaken from the <a href=\"https://arxiv.org/abs/2204.07143\">original paper</a>.</small>\n\nThis model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).\nThe original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).\n\n## Usage tips\n\n- One can use the [`AutoImageProcessor`] API to prepare images for the model.\n- NAT can be used as a *backbone*. When `output_hidden_states = True`,\nit will output both `hidden_states` and `reshaped_hidden_states`.\nThe `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than\n`(batch_size, height, width, num_channels)`.\n\nNotes:\n- NAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention.\nYou can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten),\nor build on your system by running `pip install natten`.\nNote that the latter will likely take time to compile. NATTEN does not support Windows devices yet.\n- Patch size of 4 is only supported at the moment."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 71118917-4b7b-4225-980c-585b7b23b89f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nat", "title": "Neighborhood Attention Transformer", "tokens": 1124, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"71118917-4b7b-4225-980c-585b7b23b89f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c6fbfa21-2511-49eb-8aae-62456d8ddc5a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40d20cbbd35f13e4041026c403b647a26dd23e3c4d8fde2abcbeb775a9806eee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9c9a3cd5-259b-4c38-904a-7d0bb9b713a6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cf1ddc6b638dc31f04b1cbb27e5c504fc890f3a5402dc09595c6a6bcfe51932b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1849, \"end_char_idx\": 4093, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c6fbfa21-2511-49eb-8aae-62456d8ddc5a", "doc_id": "c6fbfa21-2511-49eb-8aae-62456d8ddc5a", "ref_doc_id": "c6fbfa21-2511-49eb-8aae-62456d8ddc5a", "chroma:document": "*\n\n<img\nsrc=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/neighborhood-attention-pattern.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Neighborhood Attention compared to other attention patterns.\nTaken from the <a href=\"https://arxiv.org/abs/2204.07143\">original paper</a>.</small>\n\nThis model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).\nThe original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).\n\n## Usage tips\n\n- One can use the [`AutoImageProcessor`] API to prepare images for the model.\n- NAT can be used as a *backbone*. When `output_hidden_states = True`,\nit will output both `hidden_states` and `reshaped_hidden_states`.\nThe `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than\n`(batch_size, height, width, num_channels)`.\n\nNotes:\n- NAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention.\nYou can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten),\nor build on your system by running `pip install natten`.\nNote that the latter will likely take time to compile. NATTEN does not support Windows devices yet.\n- Patch size of 4 is only supported at the moment.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with NAT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`NatForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## NatConfig\n\n[[autodoc]] NatConfig\n\n## NatModel\n\n[[autodoc]] NatModel\n    - forward\n\n## NatForImageClassification\n\n[[autodoc]] NatForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 77d16024-518f-4743-9af0-0a1ffd137217
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"77d16024-518f-4743-9af0-0a1ffd137217\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5782943e-e5cd-4131-9582-0eb3808e9fc5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"351d593f9e99935366c8e5a8af756780e88885f5e7fde13f6a500f0d683fa36d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a9b1ecdb-ab9f-4274-9cc6-495d4942aaff\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3a1931a4ab0ad52105f36450058ff76de69a3bc0de520be2f82a5aeb0f9dc149\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2191, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5782943e-e5cd-4131-9582-0eb3808e9fc5", "doc_id": "5782943e-e5cd-4131-9582-0eb3808e9fc5", "ref_doc_id": "5782943e-e5cd-4131-9582-0eb3808e9fc5", "chroma:document": "# Depth Anything V2\n\n## Overview\n\nDepth Anything V2 was introduced in [the paper of the same name](https://arxiv.org/abs/2406.09414) by Lihe Yang et al. It uses the same architecture as the original [Depth Anything model](depth_anything), but uses synthetic data and a larger capacity teacher model to achieve much finer and robust depth predictions.\n\nThe abstract from the paper is the following:\n\n*This work presents Depth Anything V2. Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more robust depth predictions through three key practices: 1) replacing all labeled real images with synthetic images, 2) scaling up the capacity of our teacher model, and 3) teaching student models via the bridge of large-scale pseudo-labeled real images. Compared with the latest models built on Stable Diffusion, our models are significantly more efficient (more than 10x faster) and more accurate. We offer models of different scales (ranging from 25M to 1.3B params) to support extensive scenarios. Benefiting from their strong generalization capability, we fine-tune them with metric depth labels to obtain our metric depth models. In addition to our models, considering the limited diversity and frequent noise in current test sets, we construct a versatile evaluation benchmark with precise annotations and diverse scenes to facilitate future research.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Depth Anything overview. Taken from the <a href=\"https://arxiv.org/abs/2401.10891\">original paper</a>.</small>\n\nThe Depth Anything models were contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/DepthAnything/Depth-Anything-V2).\n\n## Usage example\n\nThere are 2 main ways to use Depth Anything V2: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a9b1ecdb-ab9f-4274-9cc6-495d4942aaff
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a9b1ecdb-ab9f-4274-9cc6-495d4942aaff\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5782943e-e5cd-4131-9582-0eb3808e9fc5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"351d593f9e99935366c8e5a8af756780e88885f5e7fde13f6a500f0d683fa36d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"77d16024-518f-4743-9af0-0a1ffd137217\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"74b940f36636938423d48c63695b5c5caaee9917b2794c5d459f6359bcece97d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"82b1af03-142a-4597-9940-aa8b9290a56f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e9c07136b9189b0ce9b5900b21ed8233444d62bb311fc64bd3b68152e377a8a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 438, \"end_char_idx\": 4037, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5782943e-e5cd-4131-9582-0eb3808e9fc5", "doc_id": "5782943e-e5cd-4131-9582-0eb3808e9fc5", "ref_doc_id": "5782943e-e5cd-4131-9582-0eb3808e9fc5", "chroma:document": "Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more robust depth predictions through three key practices: 1) replacing all labeled real images with synthetic images, 2) scaling up the capacity of our teacher model, and 3) teaching student models via the bridge of large-scale pseudo-labeled real images. Compared with the latest models built on Stable Diffusion, our models are significantly more efficient (more than 10x faster) and more accurate. We offer models of different scales (ranging from 25M to 1.3B params) to support extensive scenarios. Benefiting from their strong generalization capability, we fine-tune them with metric depth labels to obtain our metric depth models. In addition to our models, considering the limited diversity and frequent noise in current test sets, we construct a versatile evaluation benchmark with precise annotations and diverse scenes to facilitate future research.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Depth Anything overview. Taken from the <a href=\"https://arxiv.org/abs/2401.10891\">original paper</a>.</small>\n\nThe Depth Anything models were contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/DepthAnything/Depth-Anything-V2).\n\n## Usage example\n\nThere are 2 main ways to use Depth Anything V2: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself.\n\n### Pipeline API\n\nThe pipeline allows to use the model in a few lines of code:\n\n```python\n>>> from transformers import pipeline\n>>> from PIL import Image\n>>> import requests\n\n>>> # load pipe\n>>> pipe = pipeline(task=\"depth-estimation\", model=\"depth-anything/Depth-Anything-V2-Small-hf\")\n\n>>> # load image\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> # inference\n>>> depth = pipe(image)[\"depth\"]\n```\n\n### Using the model yourself\n\nIf you want to do the pre- and post-processing yourself, here's how to do that:\n\n```python\n>>> from transformers import AutoImageProcessor, AutoModelForDepthEstimation\n>>> import torch\n>>> import numpy as np\n>>> from PIL import Image\n>>> import requests\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\"depth-anything/Depth-Anything-V2-Small-hf\")\n>>> model = AutoModelForDepthEstimation.from_pretrained(\"depth-anything/Depth-Anything-V2-Small-hf\")\n\n>>> # prepare image for the model\n>>> inputs = image_processor(images=image, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n...     predicted_depth = outputs.predicted_depth\n\n>>> # interpolate to original size\n>>> prediction = torch.nn.functional.interpolate(\n...     predicted_depth.unsqueeze(1),\n...     size=image.size[::-1],\n...     mode=\"bicubic\",\n...     align_corners=False,\n... )\n\n>>> # visualize the prediction\n>>> output = prediction.squeeze().cpu().numpy()\n>>> formatted = (output * 255 / np.max(output)).astype(\"uint8\")\n>>> depth = Image.fromarray(formatted)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Depth Anything."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 82b1af03-142a-4597-9940-aa8b9290a56f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"82b1af03-142a-4597-9940-aa8b9290a56f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5782943e-e5cd-4131-9582-0eb3808e9fc5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"351d593f9e99935366c8e5a8af756780e88885f5e7fde13f6a500f0d683fa36d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a9b1ecdb-ab9f-4274-9cc6-495d4942aaff\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cbfc9f6cbed9dba3d61031c5318f0a97142a77187ec23a0bda55cfc61297a815\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4039, \"end_char_idx\": 4948, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5782943e-e5cd-4131-9582-0eb3808e9fc5", "doc_id": "5782943e-e5cd-4131-9582-0eb3808e9fc5", "ref_doc_id": "5782943e-e5cd-4131-9582-0eb3808e9fc5", "chroma:document": "- [Monocular depth estimation task guide](../tasks/depth_estimation)\n- [Depth Anything V2 demo](https://huggingface.co/spaces/depth-anything/Depth-Anything-V2).\n- A notebook showcasing inference with [`DepthAnythingForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Depth%20Anything/Predicting_depth_in_an_image_with_Depth_Anything.ipynb). \ud83c\udf0e\n- [Core ML conversion of the `small` variant for use on Apple Silicon](https://huggingface.co/apple/coreml-depth-anything-v2-small).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DepthAnythingConfig\n\n[[autodoc]] DepthAnythingConfig\n\n## DepthAnythingForDepthEstimation\n\n[[autodoc]] DepthAnythingForDepthEstimation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2857e5db-e149-4d35-a420-a4c64f7a0c3a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2857e5db-e149-4d35-a420-a4c64f7a0c3a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2ed49fe7-4807-4948-93ad-82cb7833e2ea\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e7a6a0c85ab1661b04e534d8a20c0e6af4dfb5a15105348102593f4fa78e4f2d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c63b4e70-bd39-4c06-b577-7fcedb11bc4e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"22770f56cc9796afcdbcaae6a9181fedc64997e178a6675226f432c5413d9aa4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3419, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "ref_doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "chroma:document": "# ALBERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=albert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-albert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/albert-base-v2\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe ALBERT model was proposed in [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942) by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma,\nRadu Soricut. It presents two parameter-reduction techniques to lower memory consumption and increase the training\nspeed of BERT:\n\n- Splitting the embedding matrix into two smaller matrices.\n- Using repeating layers split among groups.\n\nThe abstract from the paper is the following:\n\n*Increasing model size when pretraining natural language representations often results in improved performance on\ndownstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations,\nlonger training times, and unexpected model degradation. To address these problems, we present two parameter-reduction\ntechniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows\nthat our proposed methods lead to models that scale much better compared to the original BERT. We also use a\nself-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks\nwith multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and\nSQuAD benchmarks while having fewer parameters compared to BERT-large.*\n\nThis model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by\n[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).\n\n## Usage tips\n\n- ALBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather\n  than the left.\n- ALBERT uses repeating layers which results in a small memory footprint, however the computational cost remains\n  similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same\n  number of (repeating) layers.\n- Embedding size E is different from hidden size H justified because the embeddings are context independent (one embedding vector represents one token), whereas hidden states are context dependent (one hidden state represents a sequence of tokens) so it's more logical to have H >> E. Also, the embedding matrix is large since it's V x E (V being the vocab size). If E < H, it has less parameters.\n- Layers are split in groups that share parameters (to save memory).\nNext sentence prediction is replaced by a sentence ordering prediction: in the inputs, we have two sentences A and B (that are consecutive) and we either feed A followed by B or B followed by A. The model must predict if they have been swapped or not.\n\n\n\nThis model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by\n[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).\n\n\n## Resources"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c63b4e70-bd39-4c06-b577-7fcedb11bc4e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c63b4e70-bd39-4c06-b577-7fcedb11bc4e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2ed49fe7-4807-4948-93ad-82cb7833e2ea\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e7a6a0c85ab1661b04e534d8a20c0e6af4dfb5a15105348102593f4fa78e4f2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2857e5db-e149-4d35-a420-a4c64f7a0c3a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af76aeac221c99d75465927adb119fd6baa3e7774ba1778a531908ac254efb4d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7936d9e1-147f-49b2-9ced-841ee2144692\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3313f2db2e48fc4ccefa876670b462a06683db2724d3bc03cf6d0da0c79e8619\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3159, \"end_char_idx\": 6491, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "ref_doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "chroma:document": "This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by\n[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).\n\n\n## Resources\n\n\nThe resources provided in the following sections consist of a list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with AlBERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n\n- [`AlbertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification).\n\n\n- [`TFAlbertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification).\n\n- [`FlaxAlbertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- Check the [Text classification task guide](../tasks/sequence_classification) on how to use the model.\n\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n\n- [`AlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification).\n\n\n- [`TFAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n\n\n\n- [`FlaxAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Token classification task guide](../tasks/token_classification) on how to use the model.\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`AlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7936d9e1-147f-49b2-9ced-841ee2144692
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7936d9e1-147f-49b2-9ced-841ee2144692\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2ed49fe7-4807-4948-93ad-82cb7833e2ea\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e7a6a0c85ab1661b04e534d8a20c0e6af4dfb5a15105348102593f4fa78e4f2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c63b4e70-bd39-4c06-b577-7fcedb11bc4e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ced811a0e9c14d54d37d03d94a98774d7272044f5751ddbf1600456979a6ff1b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"83f19d50-466e-4694-a4ac-6e738b29a31c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9c5b42ddac3b7a3e3e0ac1ebc0874e038544a2735244f9adbb3832ba708c9736\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4838, \"end_char_idx\": 8013, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "ref_doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "chroma:document": "- [`TFAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n\n\n\n- [`FlaxAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Token classification task guide](../tasks/token_classification) on how to use the model.\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`AlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Masked language modeling task guide](../tasks/masked_language_modeling) on how to use the model.\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`AlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Question answering task guide](../tasks/question_answering) on how to use the model.\n\n**Multiple choice**\n\n- [`AlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 83f19d50-466e-4694-a4ac-6e738b29a31c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"83f19d50-466e-4694-a4ac-6e738b29a31c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2ed49fe7-4807-4948-93ad-82cb7833e2ea\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e7a6a0c85ab1661b04e534d8a20c0e6af4dfb5a15105348102593f4fa78e4f2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7936d9e1-147f-49b2-9ced-841ee2144692\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"54d8f1de37406888d761ac8d87ca5299730c59a0d44e3249b7ddd9718f22e055\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2193ea60-8852-43ff-bfe5-03f85621f49d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"816c4fae17cd9a3519904b5ec9d38fa0aaded8d39b3a74e03e0a39e09cb99646\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6492, \"end_char_idx\": 8392, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "ref_doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "chroma:document": "- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Masked language modeling task guide](../tasks/masked_language_modeling) on how to use the model.\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`AlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Question answering task guide](../tasks/question_answering) on how to use the model.\n\n**Multiple choice**\n\n- [`AlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFAlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n\n- Check the  [Multiple choice task guide](../tasks/multiple_choice) on how to use the model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2193ea60-8852-43ff-bfe5-03f85621f49d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2193ea60-8852-43ff-bfe5-03f85621f49d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2ed49fe7-4807-4948-93ad-82cb7833e2ea\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e7a6a0c85ab1661b04e534d8a20c0e6af4dfb5a15105348102593f4fa78e4f2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"83f19d50-466e-4694-a4ac-6e738b29a31c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a5fd1e13e7a49294c98e7aba38c267c71c3b93ed49ad88ba9f15ad06b5457b5d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7050, \"end_char_idx\": 10613, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "ref_doc_id": "2ed49fe7-4807-4948-93ad-82cb7833e2ea", "chroma:document": "- [`TFAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Question answering task guide](../tasks/question_answering) on how to use the model.\n\n**Multiple choice**\n\n- [`AlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFAlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n\n- Check the  [Multiple choice task guide](../tasks/multiple_choice) on how to use the model.\n\n\n## AlbertConfig\n\n[[autodoc]] AlbertConfig\n\n## AlbertTokenizer\n\n[[autodoc]] AlbertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## AlbertTokenizerFast\n\n[[autodoc]] AlbertTokenizerFast\n\n## Albert specific outputs\n\n[[autodoc]] models.albert.modeling_albert.AlbertForPreTrainingOutput\n\n[[autodoc]] models.albert.modeling_tf_albert.TFAlbertForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## AlbertModel\n\n[[autodoc]] AlbertModel\n    - forward\n\n## AlbertForPreTraining\n\n[[autodoc]] AlbertForPreTraining\n    - forward\n\n## AlbertForMaskedLM\n\n[[autodoc]] AlbertForMaskedLM\n    - forward\n\n## AlbertForSequenceClassification\n\n[[autodoc]] AlbertForSequenceClassification\n    - forward\n\n## AlbertForMultipleChoice\n\n[[autodoc]] AlbertForMultipleChoice\n\n## AlbertForTokenClassification\n\n[[autodoc]] AlbertForTokenClassification\n    - forward\n\n## AlbertForQuestionAnswering\n\n[[autodoc]] AlbertForQuestionAnswering\n    - forward\n\n</pt>\n\n<tf>\n\n## TFAlbertModel\n\n[[autodoc]] TFAlbertModel\n    - call\n\n## TFAlbertForPreTraining\n\n[[autodoc]] TFAlbertForPreTraining\n    - call\n\n## TFAlbertForMaskedLM\n\n[[autodoc]] TFAlbertForMaskedLM\n    - call\n\n## TFAlbertForSequenceClassification\n\n[[autodoc]] TFAlbertForSequenceClassification\n    - call\n\n## TFAlbertForMultipleChoice\n\n[[autodoc]] TFAlbertForMultipleChoice\n    - call\n\n## TFAlbertForTokenClassification\n\n[[autodoc]] TFAlbertForTokenClassification\n    - call\n\n## TFAlbertForQuestionAnswering\n\n[[autodoc]] TFAlbertForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxAlbertModel\n\n[[autodoc]] FlaxAlbertModel\n    - __call__\n\n## FlaxAlbertForPreTraining\n\n[[autodoc]] FlaxAlbertForPreTraining\n    - __call__\n\n## FlaxAlbertForMaskedLM\n\n[[autodoc]] FlaxAlbertForMaskedLM\n    - __call__\n\n## FlaxAlbertForSequenceClassification\n\n[[autodoc]] FlaxAlbertForSequenceClassification\n    - __call__\n\n## FlaxAlbertForMultipleChoice\n\n[[autodoc]] FlaxAlbertForMultipleChoice\n    - __call__\n\n## FlaxAlbertForTokenClassification\n\n[[autodoc]] FlaxAlbertForTokenClassification\n    - __call__\n\n## FlaxAlbertForQuestionAnswering\n\n[[autodoc]] FlaxAlbertForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 603e7065-7064-4388-8248-0e34f7999c0a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vitdet", "title": "ViTDet", "tokens": 512, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"603e7065-7064-4388-8248-0e34f7999c0a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vitdet\", \"title\": \"ViTDet\", \"tokens\": 512, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"66ddc465-c857-45db-a167-13090c791f99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vitdet\", \"title\": \"ViTDet\", \"tokens\": 512, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"01646f10dc016090b6b79c5657fcde4d5d33e6f1c6e45421abd8307cfe728e41\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1674, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "66ddc465-c857-45db-a167-13090c791f99", "doc_id": "66ddc465-c857-45db-a167-13090c791f99", "ref_doc_id": "66ddc465-c857-45db-a167-13090c791f99", "chroma:document": "# ViTDet\n\n## Overview\n\nThe ViTDet model was proposed in [Exploring Plain Vision Transformer Backbones for Object Detection](https://arxiv.org/abs/2203.16527) by Yanghao Li, Hanzi Mao, Ross Girshick, Kaiming He.\nVitDet leverages the plain [Vision Transformer](vit) for the task of object detection.\n\nThe abstract from the paper is the following:\n\n*We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone network for object detection. This design enables the original ViT architecture to be fine-tuned for object detection without needing to redesign a hierarchical backbone for pre-training. With minimal adaptations for fine-tuning, our plain-backbone detector can achieve competitive results. Surprisingly, we observe: (i) it is sufficient to build a simple feature pyramid from a single-scale feature map (without the common FPN design) and (ii) it is sufficient to use window attention (without shifting) aided with very few cross-window propagation blocks. With plain ViT backbones pre-trained as Masked Autoencoders (MAE), our detector, named ViTDet, can compete with the previous leading methods that were all based on hierarchical backbones, reaching up to 61.3 AP_box on the COCO dataset using only ImageNet-1K pre-training. We hope our study will draw attention to research on plain-backbone detectors.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet).\n\nTips:\n\n- At the moment, only the backbone is available.\n\n## VitDetConfig\n\n[[autodoc]] VitDetConfig\n\n## VitDetModel\n\n[[autodoc]] VitDetModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b74be0d6-246a-4b14-acc1-1af40a38ffea
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text", "title": "Speech2Text", "tokens": 1503, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b74be0d6-246a-4b14-acc1-1af40a38ffea\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a36172d8-6737-43e7-a1d4-00d192354327\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d1c61973c238b8052a804b1f25cc820404e5b19ca57fc1af8c321cf5db1e04d5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a0d04a60-bb34-4e34-8460-ee49899dbb22\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1aa50d67f9bdc367a17b930ac1561e6255583ee33e52e4d04aae6e0a228b9819\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2582, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a36172d8-6737-43e7-a1d4-00d192354327", "doc_id": "a36172d8-6737-43e7-a1d4-00d192354327", "ref_doc_id": "a36172d8-6737-43e7-a1d4-00d192354327", "chroma:document": "# Speech2Text\n\n## Overview\n\nThe Speech2Text model was proposed in [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://arxiv.org/abs/2010.05171) by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino. It's a\ntransformer-based seq2seq (encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech\nTranslation (ST). It uses a convolutional downsampler to reduce the length of speech inputs by 3/4th before they are\nfed into the encoder. The model is trained with standard autoregressive cross-entropy loss and generates the\ntranscripts/translations autoregressively. Speech2Text has been fine-tuned on several datasets for ASR and ST:\n[LibriSpeech](http://www.openslr.org/12), [CoVoST 2](https://github.com/facebookresearch/covost), [MuST-C](https://ict.fbk.eu/must-c/).\n\nThis model was contributed by [valhalla](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text).\n\n## Inference\n\nSpeech2Text is a speech model that accepts a float tensor of log-mel filter-bank features extracted from the speech\nsignal. It's a transformer-based seq2seq model, so the transcripts/translations are generated autoregressively. The\n`generate()` method can be used for inference.\n\nThe [`Speech2TextFeatureExtractor`] class is responsible for extracting the log-mel filter-bank\nfeatures. The [`Speech2TextProcessor`] wraps [`Speech2TextFeatureExtractor`] and\n[`Speech2TextTokenizer`] into a single instance to both extract the input features and decode the\npredicted token ids.\n\nThe feature extractor depends on `torchaudio` and the tokenizer depends on `sentencepiece` so be sure to\ninstall those packages before running the examples. You could either install those as extra speech dependencies with\n`pip install transformers\"[speech, sentencepiece]\"` or install the packages separately with `pip install torchaudio sentencepiece`. Also `torchaudio` requires the development version of the [libsndfile](http://www.mega-nerd.com/libsndfile/) package which can be installed via a system package manager. On Ubuntu it can\nbe installed as follows: `apt install libsndfile1-dev`\n\n- ASR and Speech Translation\n\n```python\n>>> import torch\n>>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> model = Speech2TextForConditionalGeneration.from_pretrained(\"facebook/s2t-small-librispeech-asr\")\n>>> processor = Speech2TextProcessor.from_pretrained(\"facebook/s2t-small-librispeech-asr\")"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a0d04a60-bb34-4e34-8460-ee49899dbb22
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text", "title": "Speech2Text", "tokens": 1503, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a0d04a60-bb34-4e34-8460-ee49899dbb22\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a36172d8-6737-43e7-a1d4-00d192354327\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d1c61973c238b8052a804b1f25cc820404e5b19ca57fc1af8c321cf5db1e04d5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b74be0d6-246a-4b14-acc1-1af40a38ffea\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d57171b4648915c7831c1922faf566e105f725249277ef7baec7b7f98bf6c843\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2585, \"end_char_idx\": 5575, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a36172d8-6737-43e7-a1d4-00d192354327", "doc_id": "a36172d8-6737-43e7-a1d4-00d192354327", "ref_doc_id": "a36172d8-6737-43e7-a1d4-00d192354327", "chroma:document": ">>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_demo\", \"clean\", split=\"validation\")\n\n>>> inputs = processor(ds[0][\"audio\"][\"array\"], sampling_rate=ds[0][\"audio\"][\"sampling_rate\"], return_tensors=\"pt\")\n>>> generated_ids = model.generate(inputs[\"input_features\"], attention_mask=inputs[\"attention_mask\"])\n\n>>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> transcription\n['mister quilter is the apostle of the middle classes and we are glad to welcome his gospel']\n```\n\n- Multilingual speech translation\n\n  For multilingual speech translation models, `eos_token_id` is used as the `decoder_start_token_id` and\n  the target language id is forced as the first generated token. To force the target language id as the first\n  generated token, pass the `forced_bos_token_id` parameter to the `generate()` method. The following\n  example shows how to transate English speech to French text using the *facebook/s2t-medium-mustc-multilingual-st*\n  checkpoint.\n\n```python\n>>> import torch\n>>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> model = Speech2TextForConditionalGeneration.from_pretrained(\"facebook/s2t-medium-mustc-multilingual-st\")\n>>> processor = Speech2TextProcessor.from_pretrained(\"facebook/s2t-medium-mustc-multilingual-st\")\n\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_demo\", \"clean\", split=\"validation\")\n\n>>> inputs = processor(ds[0][\"audio\"][\"array\"], sampling_rate=ds[0][\"audio\"][\"sampling_rate\"], return_tensors=\"pt\")\n>>> generated_ids = model.generate(\n...     inputs[\"input_features\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     forced_bos_token_id=processor.tokenizer.lang_code_to_id[\"fr\"],\n... )\n\n>>> translation = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> translation\n[\"(Vid\u00e9o) Si M. Kilder est l'apossible des classes moyennes, et nous sommes heureux d'\u00eatre accueillis dans son \u00e9vangile.\"]\n```\n\nSee the [model hub](https://huggingface.co/models?filter=speech_to_text) to look for Speech2Text checkpoints.\n\n## Speech2TextConfig\n\n[[autodoc]] Speech2TextConfig\n\n## Speech2TextTokenizer\n\n[[autodoc]] Speech2TextTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## Speech2TextFeatureExtractor\n\n[[autodoc]] Speech2TextFeatureExtractor\n    - __call__\n\n## Speech2TextProcessor\n\n[[autodoc]] Speech2TextProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n<frameworkcontent>\n<pt>\n\n## Speech2TextModel\n\n[[autodoc]] Speech2TextModel\n    - forward\n\n## Speech2TextForConditionalGeneration\n\n[[autodoc]] Speech2TextForConditionalGeneration\n    - forward\n\n</pt>\n<tf>\n\n## TFSpeech2TextModel\n\n[[autodoc]] TFSpeech2TextModel\n    - call\n\n## TFSpeech2TextForConditionalGeneration\n\n[[autodoc]] TFSpeech2TextForConditionalGeneration\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f0a144fe-91e6-47ad-9b58-025e7c42fc31
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/autoformer", "title": "Autoformer", "tokens": 739, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f0a144fe-91e6-47ad-9b58-025e7c42fc31\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/autoformer\", \"title\": \"Autoformer\", \"tokens\": 739, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cdab5895-b4d9-48b5-ace8-55f09e83b0e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/autoformer\", \"title\": \"Autoformer\", \"tokens\": 739, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"06881c14c7af9829695e303ea1c2344e27103e83e7e4ff9fb5702d97ab5f9d30\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2844, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cdab5895-b4d9-48b5-ace8-55f09e83b0e9", "doc_id": "cdab5895-b4d9-48b5-ace8-55f09e83b0e9", "ref_doc_id": "cdab5895-b4d9-48b5-ace8-55f09e83b0e9", "chroma:document": "# Autoformer\n\n## Overview\n\nThe Autoformer model was proposed in [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://arxiv.org/abs/2106.13008) by Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long.\n\nThis model augments the Transformer as a deep decomposition architecture, which can progressively decompose the trend and seasonal components during the forecasting process.\n\nThe abstract from the paper is the following:\n\n*Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease.*\n\nThis model was contributed by [elisim](https://huggingface.co/elisim) and [kashif](https://huggingface.co/kashif).\nThe original code can be found [here](https://github.com/thuml/Autoformer).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Check out the Autoformer blog-post in HuggingFace blog: [Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer)](https://huggingface.co/blog/autoformer)\n\n## AutoformerConfig\n\n[[autodoc]] AutoformerConfig\n\n## AutoformerModel\n\n[[autodoc]] AutoformerModel\n    - forward\n\n## AutoformerForPrediction\n\n[[autodoc]] AutoformerForPrediction\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9c5fc7ee-1844-4c65-89e1-0b7336865ce7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clipseg", "title": "CLIPSeg", "tokens": 1043, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9c5fc7ee-1844-4c65-89e1-0b7336865ce7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b33e8c65-805f-4da0-bbbc-5054af55b09c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef46d8407d8c0decb89a011209ad4580de1de7d0f36c541f66190201911af012\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6ebae24f-15c3-4d6f-a301-931c4f760906\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69b59b69de3d8aabb21e7fa6aae36e15c30419d1b5a6e1a3f983bdd2132ed05d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3171, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b33e8c65-805f-4da0-bbbc-5054af55b09c", "doc_id": "b33e8c65-805f-4da0-bbbc-5054af55b09c", "ref_doc_id": "b33e8c65-805f-4da0-bbbc-5054af55b09c", "chroma:document": "# CLIPSeg\n\n## Overview\n\nThe CLIPSeg model was proposed in [Image Segmentation Using Text and Image Prompts](https://arxiv.org/abs/2112.10003) by Timo L\u00fcddecke\nand Alexander Ecker. CLIPSeg adds a minimal decoder on top of a frozen [CLIP](clip) model for zero- and one-shot image segmentation.\n\nThe abstract from the paper is the following:\n\n*Image segmentation is usually addressed by training a\nmodel for a fixed set of object classes. Incorporating additional classes or more complex queries later is expensive\nas it requires re-training the model on a dataset that encompasses these expressions. Here we propose a system\nthat can generate image segmentations based on arbitrary\nprompts at test time. A prompt can be either a text or an\nimage. This approach enables us to create a unified model\n(trained once) for three common segmentation tasks, which\ncome with distinct challenges: referring expression segmentation, zero-shot segmentation and one-shot segmentation.\nWe build upon the CLIP model as a backbone which we extend with a transformer-based decoder that enables dense\nprediction. After training on an extended version of the\nPhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on\nan additional image expressing the query. We analyze different variants of the latter image-based prompts in detail.\nThis novel hybrid input allows for dynamic adaptation not\nonly to the three segmentation tasks mentioned above, but\nto any binary segmentation task where a text or image query\ncan be formulated. Finally, we find our system to adapt well\nto generalized queries involving affordances or properties*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/clipseg_architecture.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> CLIPSeg overview. Taken from the <a href=\"https://arxiv.org/abs/2112.10003\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/timojl/clipseg).\n\n## Usage tips\n\n- [`CLIPSegForImageSegmentation`] adds a decoder on top of [`CLIPSegModel`]. The latter is identical to [`CLIPModel`].\n- [`CLIPSegForImageSegmentation`] can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text\n(provided to the model as `input_ids`) or an image (provided to the model as `conditional_pixel_values`). One can also provide custom\nconditional embeddings (provided to the model as `conditional_embeddings`).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CLIPSeg. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"image-segmentation\"/>\n\n- A notebook that illustrates [zero-shot image segmentation with CLIPSeg](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/CLIPSeg/Zero_shot_image_segmentation_with_CLIPSeg.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6ebae24f-15c3-4d6f-a301-931c4f760906
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clipseg", "title": "CLIPSeg", "tokens": 1043, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6ebae24f-15c3-4d6f-a301-931c4f760906\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b33e8c65-805f-4da0-bbbc-5054af55b09c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef46d8407d8c0decb89a011209ad4580de1de7d0f36c541f66190201911af012\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9c5fc7ee-1844-4c65-89e1-0b7336865ce7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1f8fd60121a257130f51f24635c2cb76bc01c0f11a80b9ca5ca416d20452967b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1563, \"end_char_idx\": 3735, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b33e8c65-805f-4da0-bbbc-5054af55b09c", "doc_id": "b33e8c65-805f-4da0-bbbc-5054af55b09c", "ref_doc_id": "b33e8c65-805f-4da0-bbbc-5054af55b09c", "chroma:document": "Finally, we find our system to adapt well\nto generalized queries involving affordances or properties*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/clipseg_architecture.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> CLIPSeg overview. Taken from the <a href=\"https://arxiv.org/abs/2112.10003\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/timojl/clipseg).\n\n## Usage tips\n\n- [`CLIPSegForImageSegmentation`] adds a decoder on top of [`CLIPSegModel`]. The latter is identical to [`CLIPModel`].\n- [`CLIPSegForImageSegmentation`] can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text\n(provided to the model as `input_ids`) or an image (provided to the model as `conditional_pixel_values`). One can also provide custom\nconditional embeddings (provided to the model as `conditional_embeddings`).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CLIPSeg. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"image-segmentation\"/>\n\n- A notebook that illustrates [zero-shot image segmentation with CLIPSeg](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/CLIPSeg/Zero_shot_image_segmentation_with_CLIPSeg.ipynb).\n\n## CLIPSegConfig\n\n[[autodoc]] CLIPSegConfig\n    - from_text_vision_configs\n\n## CLIPSegTextConfig\n\n[[autodoc]] CLIPSegTextConfig\n\n## CLIPSegVisionConfig\n\n[[autodoc]] CLIPSegVisionConfig\n\n## CLIPSegProcessor\n\n[[autodoc]] CLIPSegProcessor\n\n## CLIPSegModel\n\n[[autodoc]] CLIPSegModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## CLIPSegTextModel\n\n[[autodoc]] CLIPSegTextModel\n    - forward\n\n## CLIPSegVisionModel\n\n[[autodoc]] CLIPSegVisionModel\n    - forward\n\n## CLIPSegForImageSegmentation\n\n[[autodoc]] CLIPSegForImageSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fa4c099c-00df-4b62-80a1-85fa4d8d6724
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/conditional_detr", "title": "Conditional DETR", "tokens": 941, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fa4c099c-00df-4b62-80a1-85fa4d8d6724\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/conditional_detr\", \"title\": \"Conditional DETR\", \"tokens\": 941, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"21c75844-028a-476f-a8fa-62369bff3c92\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/conditional_detr\", \"title\": \"Conditional DETR\", \"tokens\": 941, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b2b508072d22702b664318186d51e74a4e77481f2684e7ed24a61febf93c992a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3451, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "21c75844-028a-476f-a8fa-62369bff3c92", "doc_id": "21c75844-028a-476f-a8fa-62369bff3c92", "ref_doc_id": "21c75844-028a-476f-a8fa-62369bff3c92", "chroma:document": "# Conditional DETR\n\n## Overview\n\nThe Conditional DETR model was proposed in [Conditional DETR for Fast Training Convergence](https://arxiv.org/abs/2108.06152) by Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, Jingdong Wang. Conditional DETR presents a conditional cross-attention mechanism for fast DETR training. Conditional DETR converges 6.7\u00d7 to 10\u00d7 faster than DETR.\n\nThe abstract from the paper is the following:\n\n*The recently-developed DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings for localizing the four extremities and predicting the box, which increases the need for high-quality content embeddings and thus the training difficulty. Our approach, named conditional DETR, learns a conditional spatial query from the decoder embedding for decoder multi-head cross-attention. The benefit is that through the conditional spatial query, each cross-attention head is able to attend to a band containing a distinct region, e.g., one object extremity or a region inside the object box. This narrows down the spatial range for localizing the distinct regions for object classification and box regression, thus relaxing the dependence on the content embeddings and easing the training. Empirical results show that conditional DETR converges 6.7\u00d7 faster for the backbones R50 and R101 and 10\u00d7 faster for stronger backbones DC5-R50 and DC5-R101. Code is available at https://github.com/Atten4Vis/ConditionalDETR.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/conditional_detr_curve.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Conditional DETR shows much faster convergence compared to the original DETR. Taken from the <a href=\"https://arxiv.org/abs/2108.06152\">original paper</a>.</small>\n\nThis model was contributed by [DepuMeng](https://huggingface.co/DepuMeng). The original code can be found [here](https://github.com/Atten4Vis/ConditionalDETR).\n\n## Resources\n\n- Scripts for finetuning [`ConditionalDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\n## ConditionalDetrConfig\n\n[[autodoc]] ConditionalDetrConfig\n\n## ConditionalDetrImageProcessor\n\n[[autodoc]] ConditionalDetrImageProcessor\n    - preprocess\n    - post_process_object_detection\n    - post_process_instance_segmentation\n    - post_process_semantic_segmentation\n    - post_process_panoptic_segmentation\n\n## ConditionalDetrFeatureExtractor\n\n[[autodoc]] ConditionalDetrFeatureExtractor\n    - __call__\n    - post_process_object_detection\n    - post_process_instance_segmentation\n    - post_process_semantic_segmentation\n    - post_process_panoptic_segmentation\n\n## ConditionalDetrModel\n\n[[autodoc]] ConditionalDetrModel\n    - forward\n\n## ConditionalDetrForObjectDetection\n\n[[autodoc]] ConditionalDetrForObjectDetection\n    - forward\n\n## ConditionalDetrForSegmentation\n\n[[autodoc]] ConditionalDetrForSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f2530282-455d-417b-bfd0-8296a8a4a62f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/visual_bert", "title": "VisualBERT", "tokens": 1381, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f2530282-455d-417b-bfd0-8296a8a4a62f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4eb16caa-4ebb-4115-b11d-b17e54abae13\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4fb9b8eb6547837d3757dfd85c512c002f2927921619ba9c8d332b96e9f3f319\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1c1e7331-20fa-4c78-8ed7-b957268d9324\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a09b812c6f15e320381f11029ec11514c916c322db8618c98a1de869df74b38b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3487, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4eb16caa-4ebb-4115-b11d-b17e54abae13", "doc_id": "4eb16caa-4ebb-4115-b11d-b17e54abae13", "ref_doc_id": "4eb16caa-4ebb-4115-b11d-b17e54abae13", "chroma:document": "# VisualBERT\n\n## Overview\n\nThe VisualBERT model was proposed in [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://arxiv.org/pdf/1908.03557) by Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang.\nVisualBERT is a neural network trained on a variety of (image, text) pairs.\n\nThe abstract from the paper is the following:\n\n*We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks.\nVisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an\nassociated input image with self-attention. We further propose two visually-grounded language model objectives for\npre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2,\nand Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly\nsimpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any\nexplicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between\nverbs and image regions corresponding to their arguments.*\n\nThis model was contributed by [gchhablani](https://huggingface.co/gchhablani). The original code can be found [here](https://github.com/uclanlp/visualbert).\n\n## Usage tips\n\n1. Most of the checkpoints provided work with the [`VisualBertForPreTraining`] configuration. Other\n   checkpoints provided are the fine-tuned checkpoints for down-stream tasks - VQA ('visualbert-vqa'), VCR\n   ('visualbert-vcr'), NLVR2 ('visualbert-nlvr2'). Hence, if you are not working on these downstream tasks, it is\n   recommended that you use the pretrained checkpoints.\n\n2. For the VCR task, the authors use a fine-tuned detector for generating visual embeddings, for all the checkpoints.\n   We do not provide the detector and its weights as a part of the package, but it will be available in the research\n   projects, and the states can be loaded directly into the detector provided.\n\nVisualBERT is a multi-modal vision and language model. It can be used for visual question answering, multiple choice,\nvisual reasoning and region-to-phrase correspondence tasks. VisualBERT uses a BERT-like transformer to prepare\nembeddings for image-text pairs. Both the text and visual features are then projected to a latent space with identical\ndimension.\n\nTo feed images to the model, each image is passed through a pre-trained object detector and the regions and the\nbounding boxes are extracted. The authors use the features generated after passing these regions through a pre-trained\nCNN like ResNet as visual embeddings. They also add absolute position embeddings, and feed the resulting sequence of\nvectors to a standard BERT model. The text input is concatenated in the front of the visual embeddings in the embedding\nlayer, and is expected to be bound by [CLS] and a [SEP] tokens, as in BERT. The segment IDs must also be set\nappropriately for the textual and visual parts.\n\nThe [`BertTokenizer`] is used to encode the text. A custom detector/image processor must be used\nto get the visual embeddings. The following example notebooks show how to use VisualBERT with Detectron-like models:\n\n- [VisualBERT VQA demo notebook](https://github.com/huggingface/transformers/tree/main/examples/research_projects/visual_bert) : This notebook\n  contains an example on VisualBERT VQA."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1c1e7331-20fa-4c78-8ed7-b957268d9324
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/visual_bert", "title": "VisualBERT", "tokens": 1381, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1c1e7331-20fa-4c78-8ed7-b957268d9324\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4eb16caa-4ebb-4115-b11d-b17e54abae13\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4fb9b8eb6547837d3757dfd85c512c002f2927921619ba9c8d332b96e9f3f319\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f2530282-455d-417b-bfd0-8296a8a4a62f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"998c17b970633b808c34a8747240cc978dc11f8d448df3aa232d703ea3348bf8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aea661bc-05f6-482d-bbb3-9b6b1bbc9769\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"82ae1ddef447329ff73792fbbc2a5900dc8a64b2a29d3f35fbc88178c249e8c5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1668, \"end_char_idx\": 4092, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4eb16caa-4ebb-4115-b11d-b17e54abae13", "doc_id": "4eb16caa-4ebb-4115-b11d-b17e54abae13", "ref_doc_id": "4eb16caa-4ebb-4115-b11d-b17e54abae13", "chroma:document": "Hence, if you are not working on these downstream tasks, it is\n   recommended that you use the pretrained checkpoints.\n\n2. For the VCR task, the authors use a fine-tuned detector for generating visual embeddings, for all the checkpoints.\n   We do not provide the detector and its weights as a part of the package, but it will be available in the research\n   projects, and the states can be loaded directly into the detector provided.\n\nVisualBERT is a multi-modal vision and language model. It can be used for visual question answering, multiple choice,\nvisual reasoning and region-to-phrase correspondence tasks. VisualBERT uses a BERT-like transformer to prepare\nembeddings for image-text pairs. Both the text and visual features are then projected to a latent space with identical\ndimension.\n\nTo feed images to the model, each image is passed through a pre-trained object detector and the regions and the\nbounding boxes are extracted. The authors use the features generated after passing these regions through a pre-trained\nCNN like ResNet as visual embeddings. They also add absolute position embeddings, and feed the resulting sequence of\nvectors to a standard BERT model. The text input is concatenated in the front of the visual embeddings in the embedding\nlayer, and is expected to be bound by [CLS] and a [SEP] tokens, as in BERT. The segment IDs must also be set\nappropriately for the textual and visual parts.\n\nThe [`BertTokenizer`] is used to encode the text. A custom detector/image processor must be used\nto get the visual embeddings. The following example notebooks show how to use VisualBERT with Detectron-like models:\n\n- [VisualBERT VQA demo notebook](https://github.com/huggingface/transformers/tree/main/examples/research_projects/visual_bert) : This notebook\n  contains an example on VisualBERT VQA.\n\n- [Generate Embeddings for VisualBERT (Colab Notebook)](https://colab.research.google.com/drive/1bLGxKdldwqnMVA5x4neY7-l_8fKGWQYI?usp=sharing) : This notebook contains\n  an example on how to generate visual embeddings.\n\nThe following example shows how to get the last hidden state using [`VisualBertModel`]:\n\n```python\n>>> import torch\n>>> from transformers import BertTokenizer, VisualBertModel\n\n>>> model = VisualBertModel.from_pretrained(\"uclanlp/visualbert-vqa-coco-pre\")\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n\n>>> inputs = tokenizer(\"What is the man eating?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: aea661bc-05f6-482d-bbb3-9b6b1bbc9769
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/visual_bert", "title": "VisualBERT", "tokens": 1381, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"aea661bc-05f6-482d-bbb3-9b6b1bbc9769\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4eb16caa-4ebb-4115-b11d-b17e54abae13\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4fb9b8eb6547837d3757dfd85c512c002f2927921619ba9c8d332b96e9f3f319\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c1e7331-20fa-4c78-8ed7-b957268d9324\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a3083602ba76ee237baace655f991ddc9c2f3fc77bf32259cdf03174f6e47b31\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2463, \"end_char_idx\": 5287, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4eb16caa-4ebb-4115-b11d-b17e54abae13", "doc_id": "4eb16caa-4ebb-4115-b11d-b17e54abae13", "ref_doc_id": "4eb16caa-4ebb-4115-b11d-b17e54abae13", "chroma:document": "To feed images to the model, each image is passed through a pre-trained object detector and the regions and the\nbounding boxes are extracted. The authors use the features generated after passing these regions through a pre-trained\nCNN like ResNet as visual embeddings. They also add absolute position embeddings, and feed the resulting sequence of\nvectors to a standard BERT model. The text input is concatenated in the front of the visual embeddings in the embedding\nlayer, and is expected to be bound by [CLS] and a [SEP] tokens, as in BERT. The segment IDs must also be set\nappropriately for the textual and visual parts.\n\nThe [`BertTokenizer`] is used to encode the text. A custom detector/image processor must be used\nto get the visual embeddings. The following example notebooks show how to use VisualBERT with Detectron-like models:\n\n- [VisualBERT VQA demo notebook](https://github.com/huggingface/transformers/tree/main/examples/research_projects/visual_bert) : This notebook\n  contains an example on VisualBERT VQA.\n\n- [Generate Embeddings for VisualBERT (Colab Notebook)](https://colab.research.google.com/drive/1bLGxKdldwqnMVA5x4neY7-l_8fKGWQYI?usp=sharing) : This notebook contains\n  an example on how to generate visual embeddings.\n\nThe following example shows how to get the last hidden state using [`VisualBertModel`]:\n\n```python\n>>> import torch\n>>> from transformers import BertTokenizer, VisualBertModel\n\n>>> model = VisualBertModel.from_pretrained(\"uclanlp/visualbert-vqa-coco-pre\")\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n\n>>> inputs = tokenizer(\"What is the man eating?\", return_tensors=\"pt\")\n>>> # this is a custom function that returns the visual embeddings given the image path\n>>> visual_embeds = get_visual_embeddings(image_path)\n\n>>> visual_token_type_ids = torch.ones(visual_embeds.shape[:-1], dtype=torch.long)\n>>> visual_attention_mask = torch.ones(visual_embeds.shape[:-1], dtype=torch.float)\n>>> inputs.update(\n...     {\n...         \"visual_embeds\": visual_embeds,\n...         \"visual_token_type_ids\": visual_token_type_ids,\n...         \"visual_attention_mask\": visual_attention_mask,\n...     }\n... )\n>>> outputs = model(**inputs)\n>>> last_hidden_state = outputs.last_hidden_state\n```\n\n## VisualBertConfig\n\n[[autodoc]] VisualBertConfig\n\n## VisualBertModel\n\n[[autodoc]] VisualBertModel\n    - forward\n\n## VisualBertForPreTraining\n\n[[autodoc]] VisualBertForPreTraining\n    - forward\n\n## VisualBertForQuestionAnswering\n\n[[autodoc]] VisualBertForQuestionAnswering\n    - forward\n\n## VisualBertForMultipleChoice\n\n[[autodoc]] VisualBertForMultipleChoice\n    - forward\n\n## VisualBertForVisualReasoning\n\n[[autodoc]] VisualBertForVisualReasoning\n    - forward\n\n## VisualBertForRegionToPhraseAlignment\n\n[[autodoc]] VisualBertForRegionToPhraseAlignment\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: afb01585-6aa1-4a8e-9ab3-53328ff29b7e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus", "title": "BigBirdPegasus", "tokens": 1076, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"afb01585-6aa1-4a8e-9ab3-53328ff29b7e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1cb53f87-9ff7-42d8-898d-efa474613939\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87240467373d46094aa3de268a3cea6ca1a339a55cd83f79bb00188787e5e76a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0e7784e9-c823-4861-981a-361f8c3980b8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"353bd4fcf3a1e63f7c95dc6d74428844dbad4626a6eea493c5bd74fffef0656e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3099, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1cb53f87-9ff7-42d8-898d-efa474613939", "doc_id": "1cb53f87-9ff7-42d8-898d-efa474613939", "ref_doc_id": "1cb53f87-9ff7-42d8-898d-efa474613939", "chroma:document": "# BigBirdPegasus\n\n## Overview\n\nThe BigBird model was proposed in [Big Bird: Transformers for Longer Sequences](https://arxiv.org/abs/2007.14062) by\nZaheer, Manzil and Guruganesh, Guru and Dubey, Kumar Avinava and Ainslie, Joshua and Alberti, Chris and Ontanon,\nSantiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and others. BigBird, is a sparse-attention\nbased transformer which extends Transformer based models, such as BERT to much longer sequences. In addition to sparse\nattention, BigBird also applies global attention as well as random attention to the input sequence. Theoretically, it\nhas been shown that applying sparse, global, and random attention approximates full attention, while being\ncomputationally much more efficient for longer sequences. As a consequence of the capability to handle longer context,\nBigBird has shown improved performance on various long document NLP tasks, such as question answering and\nsummarization, compared to BERT or RoBERTa.\n\nThe abstract from the paper is the following:\n\n*Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP.\nUnfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence\nlength due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that\nreduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and\nis Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our\ntheoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire\nsequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to\n8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context,\nBigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also\npropose novel applications to genomics data.*\n\nThe original code can be found [here](https://github.com/google-research/bigbird).\n\n## Usage tips\n\n- For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird).\n- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using\n  **original_full** is advised as there is no benefit in using **block_sparse** attention.\n- The code currently uses window size of 3 blocks and 2 global blocks.\n- Sequence length must be divisible by block size.\n- Current implementation supports only **ITC**.\n- Current implementation doesn't support **num_random_blocks = 0**.\n- BigBirdPegasus uses the [PegasusTokenizer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pegasus/tokenization_pegasus.py).\n- BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0e7784e9-c823-4861-981a-361f8c3980b8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus", "title": "BigBirdPegasus", "tokens": 1076, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0e7784e9-c823-4861-981a-361f8c3980b8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1cb53f87-9ff7-42d8-898d-efa474613939\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87240467373d46094aa3de268a3cea6ca1a339a55cd83f79bb00188787e5e76a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"afb01585-6aa1-4a8e-9ab3-53328ff29b7e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b535d242e94bf733df2ff0dce05f2dcc1835fd1d7aab76fc84995808af027cfd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1312, \"end_char_idx\": 3960, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1cb53f87-9ff7-42d8-898d-efa474613939", "doc_id": "1cb53f87-9ff7-42d8-898d-efa474613939", "ref_doc_id": "1cb53f87-9ff7-42d8-898d-efa474613939", "chroma:document": "To remedy this, we propose, BigBird, a sparse attention mechanism that\nreduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and\nis Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our\ntheoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire\nsequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to\n8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context,\nBigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also\npropose novel applications to genomics data.*\n\nThe original code can be found [here](https://github.com/google-research/bigbird).\n\n## Usage tips\n\n- For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird).\n- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using\n  **original_full** is advised as there is no benefit in using **block_sparse** attention.\n- The code currently uses window size of 3 blocks and 2 global blocks.\n- Sequence length must be divisible by block size.\n- Current implementation supports only **ITC**.\n- Current implementation doesn't support **num_random_blocks = 0**.\n- BigBirdPegasus uses the [PegasusTokenizer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pegasus/tokenization_pegasus.py).\n- BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## BigBirdPegasusConfig\n\n[[autodoc]] BigBirdPegasusConfig\n    - all\n\n## BigBirdPegasusModel\n\n[[autodoc]] BigBirdPegasusModel\n    - forward\n\n## BigBirdPegasusForConditionalGeneration\n\n[[autodoc]] BigBirdPegasusForConditionalGeneration\n    - forward\n\n## BigBirdPegasusForSequenceClassification\n\n[[autodoc]] BigBirdPegasusForSequenceClassification\n    - forward\n\n## BigBirdPegasusForQuestionAnswering\n\n[[autodoc]] BigBirdPegasusForQuestionAnswering\n    - forward\n\n## BigBirdPegasusForCausalLM\n\n[[autodoc]] BigBirdPegasusForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: aec33220-a945-4496-bc3c-89c6bb753626
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/efficientnet", "title": "EfficientNet", "tokens": 631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"aec33220-a945-4496-bc3c-89c6bb753626\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientnet\", \"title\": \"EfficientNet\", \"tokens\": 631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"aef8fbdb-78f4-4921-859a-e0f39d4ba94c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientnet\", \"title\": \"EfficientNet\", \"tokens\": 631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e5dee94934426995ab584bc3293758e91ef68a0195e97ddf0e941a2c2bd0e3b0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2129, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "aef8fbdb-78f4-4921-859a-e0f39d4ba94c", "doc_id": "aef8fbdb-78f4-4921-859a-e0f39d4ba94c", "ref_doc_id": "aef8fbdb-78f4-4921-859a-e0f39d4ba94c", "chroma:document": "# EfficientNet\n\n## Overview\n\nThe EfficientNet model was proposed in [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://arxiv.org/abs/1905.11946) \nby Mingxing Tan and Quoc V. Le. EfficientNets are a family of image classification models, which achieve state-of-the-art accuracy, yet being an order-of-magnitude smaller and faster than previous models.\n\nThe abstract from the paper is the following:\n\n*Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.\nTo go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters.*\n\nThis model was contributed by [adirik](https://huggingface.co/adirik).\nThe original code can be found [here](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet).\n\n\n## EfficientNetConfig\n\n[[autodoc]] EfficientNetConfig\n\n## EfficientNetImageProcessor\n\n[[autodoc]] EfficientNetImageProcessor\n    - preprocess\n\n## EfficientNetModel\n\n[[autodoc]] EfficientNetModel\n    - forward\n\n## EfficientNetForImageClassification\n\n[[autodoc]] EfficientNetForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a8a7b27a-3ee6-4ee4-8910-a555607858ca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flan-ul2", "title": "FLAN-UL2", "tokens": 675, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a8a7b27a-3ee6-4ee4-8910-a555607858ca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flan-ul2\", \"title\": \"FLAN-UL2\", \"tokens\": 675, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3fbe8ead-9bb7-4e1d-8886-7c1fa5334683\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flan-ul2\", \"title\": \"FLAN-UL2\", \"tokens\": 675, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f737f3701ceef3e224cfd2e1bb77f364fedd8528bddf4e5e64f4946ce8b052b7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2170, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3fbe8ead-9bb7-4e1d-8886-7c1fa5334683", "doc_id": "3fbe8ead-9bb7-4e1d-8886-7c1fa5334683", "ref_doc_id": "3fbe8ead-9bb7-4e1d-8886-7c1fa5334683", "chroma:document": "# FLAN-UL2\n\n## Overview\n\nFlan-UL2 is an encoder decoder model based on the T5 architecture. It uses the same configuration as the [UL2](ul2) model released earlier last year. \nIt was fine tuned using the \"Flan\" prompt tuning and dataset collection. Similar to `Flan-T5`,  one can directly use FLAN-UL2 weights without finetuning the model:\n\nAccording to the original blog here are the notable improvements:\n\n- The original UL2 model was only trained with receptive field of 512, which made it non-ideal for N-shot prompting where N is large.\n- The Flan-UL2 checkpoint uses a receptive field of 2048 which makes it more usable for few-shot in-context learning.\n- The original UL2 model also had mode switch tokens that was rather mandatory to get good performance. However, they were a little cumbersome as this requires often some changes during inference or finetuning. In this update/change, we continue training UL2 20B for an additional 100k steps (with small batch) to forget \u201cmode tokens\u201d before applying Flan instruction tuning. This Flan-UL2 checkpoint does not require mode tokens anymore.\nGoogle has released the following variants:\n\nThe original checkpoints can be found [here](https://github.com/google-research/t5x/blob/main/docs/models.md#flan-ul2-checkpoints).\n\n\n## Running on low resource devices\n\nThe model is pretty heavy (~40GB in half precision) so if you just want to run the model, make sure you load your model in 8bit, and use `device_map=\"auto\"` to make sure  you don't have any OOM issue!\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/flan-ul2\", load_in_8bit=True, device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/flan-ul2\")\n\n>>> inputs = tokenizer(\"A step by step recipe to make bolognese pasta:\", return_tensors=\"pt\")\n>>> outputs = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['In a large skillet, brown the ground beef and onion over medium heat. Add the garlic']\n```\n\n<Tip>\n\nRefer to [T5's documentation page](t5) for API reference, tips, code examples and notebooks. \n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ef145231-574f-42e4-885f-125c26a6966e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nougat", "title": "Nougat", "tokens": 1239, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ef145231-574f-42e4-885f-125c26a6966e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9404d7b5-8274-408e-8308-062218ab21e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8bbe6029aaa6134aa003a2b5d82b1266877a637ed7fa516f6ba0714175b8b2f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"db5b98c2-e9a0-48de-acbe-7227386ce817\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"566a7aba6b86094d47703f50a0e286c15a07946e9ed9751428c39be649d81395\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2716, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9404d7b5-8274-408e-8308-062218ab21e9", "doc_id": "9404d7b5-8274-408e-8308-062218ab21e9", "ref_doc_id": "9404d7b5-8274-408e-8308-062218ab21e9", "chroma:document": "# Nougat\n\n## Overview\n\nThe Nougat model was proposed in [Nougat: Neural Optical Understanding for Academic Documents](https://arxiv.org/abs/2308.13418) by\nLukas Blecher, Guillem Cucurull, Thomas Scialom, Robert Stojnic. Nougat uses the same architecture as [Donut](donut), meaning an image Transformer\nencoder and an autoregressive text Transformer decoder to translate scientific PDFs to markdown, enabling easier access to them.\n\nThe abstract from the paper is the following:\n\n*Scientific knowledge is predominantly stored in books and scientific journals, often in the form of PDFs. However, the PDF format leads to a loss of semantic information, particularly for mathematical expressions. We propose Nougat (Neural Optical Understanding for Academic Documents), a Visual Transformer model that performs an Optical Character Recognition (OCR) task for processing scientific documents into a markup language, and demonstrate the effectiveness of our model on a new dataset of scientific documents. The proposed approach offers a promising solution to enhance the accessibility of scientific knowledge in the digital age, by bridging the gap between human-readable documents and machine-readable text. We release the models and code to accelerate future work on scientific text recognition.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/nougat_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Nougat high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2308.13418\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/facebookresearch/nougat).\n\n## Usage tips\n\n- The quickest way to get started with Nougat is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Nougat), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- Nougat is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework. The model is identical to [Donut](donut) in terms of architecture.\n\n## Inference\n\nNougat's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`NougatImageProcessor`] class is responsible for preprocessing the input image and\n[`NougatTokenizerFast`] decodes the generated target tokens to the target string. The\n[`NougatProcessor`] wraps [`NougatImageProcessor`] and [`NougatTokenizerFast`] classes\ninto a single instance to both extract the input features and decode the predicted token ids."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: db5b98c2-e9a0-48de-acbe-7227386ce817
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nougat", "title": "Nougat", "tokens": 1239, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"db5b98c2-e9a0-48de-acbe-7227386ce817\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9404d7b5-8274-408e-8308-062218ab21e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8bbe6029aaa6134aa003a2b5d82b1266877a637ed7fa516f6ba0714175b8b2f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ef145231-574f-42e4-885f-125c26a6966e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"97fcfc43c8fdc6e096f13157cded6165dbdc44ab73b5cc839c63213e34cba399\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2b8fa524-f81f-4078-bb3d-6ca452744fac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ff184a9927f33c2670b373752f1b149dbeed551acb98ac6f8a839f46df964899\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1204, \"end_char_idx\": 4342, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9404d7b5-8274-408e-8308-062218ab21e9", "doc_id": "9404d7b5-8274-408e-8308-062218ab21e9", "ref_doc_id": "9404d7b5-8274-408e-8308-062218ab21e9", "chroma:document": "We release the models and code to accelerate future work on scientific text recognition.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/nougat_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Nougat high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2308.13418\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/facebookresearch/nougat).\n\n## Usage tips\n\n- The quickest way to get started with Nougat is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Nougat), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- Nougat is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework. The model is identical to [Donut](donut) in terms of architecture.\n\n## Inference\n\nNougat's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`NougatImageProcessor`] class is responsible for preprocessing the input image and\n[`NougatTokenizerFast`] decodes the generated target tokens to the target string. The\n[`NougatProcessor`] wraps [`NougatImageProcessor`] and [`NougatTokenizerFast`] classes\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step PDF transcription\n\n```py\n>>> from huggingface_hub import hf_hub_download\n>>> import re\n>>> from PIL import Image\n\n>>> from transformers import NougatProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = NougatProcessor.from_pretrained(\"facebook/nougat-base\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"facebook/nougat-base\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # prepare PDF image for the model\n>>> filepath = hf_hub_download(repo_id=\"hf-internal-testing/fixtures_docvqa\", filename=\"nougat_paper.png\", repo_type=\"dataset\")\n>>> image = Image.open(filepath)\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> # generate transcription (here we only generate 30 tokens)\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     min_length=1,\n...     max_new_tokens=30,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n... )\n\n>>> sequence = processor.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> sequence = processor.post_process_generation(sequence, fix_markdown=False)\n>>> # note: we're using repr here such for the sake of printing the \\n characters, feel free to just print the sequence\n>>> print(repr(sequence))\n'\\n\\n# Nougat: Neural Optical Understanding for Academic Documents\\n\\n Lukas Blecher\\n\\nCorrespondence to: lblecher@'\n```\n\nSee the [model hub](https://huggingface.co/models?filter=nougat) to look for Nougat checkpoints.\n\n<Tip>\n\nThe model is identical to [Donut](donut) in terms of architecture."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2b8fa524-f81f-4078-bb3d-6ca452744fac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nougat", "title": "Nougat", "tokens": 1239, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2b8fa524-f81f-4078-bb3d-6ca452744fac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9404d7b5-8274-408e-8308-062218ab21e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8bbe6029aaa6134aa003a2b5d82b1266877a637ed7fa516f6ba0714175b8b2f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"db5b98c2-e9a0-48de-acbe-7227386ce817\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d717b2e4e1707b3cf5d0ecf64f1252f97f5edbbb6ac996f7a567995ed6b9fd43\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2718, \"end_char_idx\": 4653, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9404d7b5-8274-408e-8308-062218ab21e9", "doc_id": "9404d7b5-8274-408e-8308-062218ab21e9", "ref_doc_id": "9404d7b5-8274-408e-8308-062218ab21e9", "chroma:document": "- Step-by-step PDF transcription\n\n```py\n>>> from huggingface_hub import hf_hub_download\n>>> import re\n>>> from PIL import Image\n\n>>> from transformers import NougatProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = NougatProcessor.from_pretrained(\"facebook/nougat-base\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"facebook/nougat-base\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # prepare PDF image for the model\n>>> filepath = hf_hub_download(repo_id=\"hf-internal-testing/fixtures_docvqa\", filename=\"nougat_paper.png\", repo_type=\"dataset\")\n>>> image = Image.open(filepath)\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> # generate transcription (here we only generate 30 tokens)\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     min_length=1,\n...     max_new_tokens=30,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n... )\n\n>>> sequence = processor.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> sequence = processor.post_process_generation(sequence, fix_markdown=False)\n>>> # note: we're using repr here such for the sake of printing the \\n characters, feel free to just print the sequence\n>>> print(repr(sequence))\n'\\n\\n# Nougat: Neural Optical Understanding for Academic Documents\\n\\n Lukas Blecher\\n\\nCorrespondence to: lblecher@'\n```\n\nSee the [model hub](https://huggingface.co/models?filter=nougat) to look for Nougat checkpoints.\n\n<Tip>\n\nThe model is identical to [Donut](donut) in terms of architecture.\n\n</Tip>\n\n## NougatImageProcessor\n\n[[autodoc]] NougatImageProcessor\n    - preprocess\n\n## NougatTokenizerFast\n\n[[autodoc]] NougatTokenizerFast\n\n## NougatProcessor\n\n[[autodoc]] NougatProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n    - post_process_generation"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e2a4b955-4d26-4882-9d14-2c5859b741d3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava", "title": "LLaVa", "tokens": 1603, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e2a4b955-4d26-4882-9d14-2c5859b741d3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"24ba8abf-d980-488e-986c-c7b2aa15311a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b701c908513dd5f996680f0eae53b48d701dd7837603dd1b256995c9272a0db0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e4ac4d05-6979-4659-81ae-311c992ca3cc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7f86e95301b0fa84e31546635c55a3d2098008eae10dedaf1e0d6c2a53263153\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3120, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "doc_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "ref_doc_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "chroma:document": "# LLaVa\n\n## Overview\n\nLLaVa is an open-source chatbot trained by fine-tuning LlamA/Vicuna on GPT-generated multimodal instruction-following data. It is an auto-regressive language model, based on the transformer architecture. In other words, it is an multi-modal version of LLMs fine-tuned for chat / instructions.\n\nThe LLaVa model was proposed in [Visual Instruction Tuning](https://arxiv.org/abs/2304.08485) and improved in [Improved Baselines with Visual Instruction Tuning](https://arxiv.org/pdf/2310.03744) by Haotian Liu, Chunyuan Li, Yuheng Li and Yong Jae Lee.\n\nThe abstract from the paper is the following:\n\n*Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in \u223c1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/llava_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> LLaVa architecture. Taken from the <a href=\"https://arxiv.org/abs/2304.08485\">original paper.</a> </small>\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ) and [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/main/llava).\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities, as follows:\n\n```python\nfrom transformers import AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/llava-1.5-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e4ac4d05-6979-4659-81ae-311c992ca3cc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava", "title": "LLaVa", "tokens": 1603, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e4ac4d05-6979-4659-81ae-311c992ca3cc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"24ba8abf-d980-488e-986c-c7b2aa15311a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b701c908513dd5f996680f0eae53b48d701dd7837603dd1b256995c9272a0db0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e2a4b955-4d26-4882-9d14-2c5859b741d3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3cc359b1e28c6705cc0ddaec20ce9d67537f9a25e10c4146f821aee41d2ad687\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"45624936-81d5-4a5b-8f03-9dd70ebe7318\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ddfdd40c163088f405f669c5eab9c957d2a5fb44425db85c0b2b0e7e5dbf4520\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1552, \"end_char_idx\": 3649, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "doc_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "ref_doc_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2304.08485\">original paper.</a> </small>\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ) and [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/main/llava).\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities, as follows:\n\n```python\nfrom transformers import AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/llava-1.5-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"USER: <image>\\n<What\u2019s shown in this image? ASSISTANT: This image shows a red stop sign.</s>USER: Describe the image in more details."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 45624936-81d5-4a5b-8f03-9dd70ebe7318
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava", "title": "LLaVa", "tokens": 1603, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"45624936-81d5-4a5b-8f03-9dd70ebe7318\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"24ba8abf-d980-488e-986c-c7b2aa15311a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b701c908513dd5f996680f0eae53b48d701dd7837603dd1b256995c9272a0db0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4ac4d05-6979-4659-81ae-311c992ca3cc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ed49a22df1fef11c4ec34f126e1c318c4ead72024713b68af5be4322296d4e42\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7fae6880-955c-4eb0-bba8-a96b6524b216\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"029d74141add494a0cc659b8bb676883fb4c9cd5053a1f2c0f8945cd8eb4b9d4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1989, \"end_char_idx\": 4578, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "doc_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "ref_doc_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "chroma:document": "Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities, as follows:\n\n```python\nfrom transformers import AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/llava-1.5-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"USER: <image>\\n<What\u2019s shown in this image? ASSISTANT: This image shows a red stop sign.</s>USER: Describe the image in more details. ASSISTANT:\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of prompt formats accepted by each llava checkpoint:\n\n[llava-interleave models](https://huggingface.co/collections/llava-hf/llava-interleave-668e19a97da0036aad4a2f19) requires the following format:\n```bash\n\"<|im_start|>user <image>\\nWhat is shown in this image?<|im_end|><|im_start|>assistant\"\n```\n\nFor multiple turns conversation:\n\n```bash\n\"<|im_start|>user <image>\\n<prompt1><|im_end|><|im_start|>assistant <answer1><|im_end|><|im_start|>user <image>\\n<prompt1><|im_end|><|im_start|>assistant \"\n```\n\n[llava-1.5 models](https://huggingface.co/collections/llava-hf/llava-15-65f762d5b6941db5c2ba07e0) requires the following format:\n```bash\n\"USER: <image>\\n<prompt> ASSISTANT:\"\n```\n\nFor multiple turns conversation:\n\n```bash\n\"USER: <image>\\n<prompt1> ASSISTANT: <answer1></s>USER: <prompt2> ASSISTANT: <answer2></s>USER: <prompt3> ASSISTANT:\"\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7fae6880-955c-4eb0-bba8-a96b6524b216
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava", "title": "LLaVa", "tokens": 1603, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7fae6880-955c-4eb0-bba8-a96b6524b216\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"24ba8abf-d980-488e-986c-c7b2aa15311a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b701c908513dd5f996680f0eae53b48d701dd7837603dd1b256995c9272a0db0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45624936-81d5-4a5b-8f03-9dd70ebe7318\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"549e10a71c56a2e1e70ffddcf8422d07f7c7157cb6950382865198c6493727c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3257, \"end_char_idx\": 5545, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "doc_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "ref_doc_id": "24ba8abf-d980-488e-986c-c7b2aa15311a", "chroma:document": "\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"USER: <image>\\n<What\u2019s shown in this image? ASSISTANT: This image shows a red stop sign.</s>USER: Describe the image in more details. ASSISTANT:\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of prompt formats accepted by each llava checkpoint:\n\n[llava-interleave models](https://huggingface.co/collections/llava-hf/llava-interleave-668e19a97da0036aad4a2f19) requires the following format:\n```bash\n\"<|im_start|>user <image>\\nWhat is shown in this image?<|im_end|><|im_start|>assistant\"\n```\n\nFor multiple turns conversation:\n\n```bash\n\"<|im_start|>user <image>\\n<prompt1><|im_end|><|im_start|>assistant <answer1><|im_end|><|im_start|>user <image>\\n<prompt1><|im_end|><|im_start|>assistant \"\n```\n\n[llava-1.5 models](https://huggingface.co/collections/llava-hf/llava-15-65f762d5b6941db5c2ba07e0) requires the following format:\n```bash\n\"USER: <image>\\n<prompt> ASSISTANT:\"\n```\n\nFor multiple turns conversation:\n\n```bash\n\"USER: <image>\\n<prompt1> ASSISTANT: <answer1></s>USER: <prompt2> ASSISTANT: <answer2></s>USER: <prompt3> ASSISTANT:\"\n```\n\n\n### Using Flash Attention 2\n\nFlash Attention 2 is an even faster, optimized version of the previous optimization, please refer to the [Flash Attention 2 section of performance docs](https://huggingface.co/docs/transformers/perf_infer_gpu_one).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BEiT.\n\n<PipelineTag pipeline=\"image-to-text\"/>\n\n- A [Google Colab demo](https://colab.research.google.com/drive/1qsl6cd2c8gGtEW1xV5io7S8NHh-Cp1TV?usp=sharing) on how to run Llava on a free-tier Google colab instance leveraging 4-bit inference.\n- A [similar notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LLaVa/Inference_with_LLaVa_for_multimodal_generation.ipynb) showcasing batched inference. \ud83c\udf0e\n\n\n## LlavaConfig\n\n[[autodoc]] LlavaConfig\n\n## LlavaProcessor\n\n[[autodoc]] LlavaProcessor\n\n## LlavaForConditionalGeneration\n\n[[autodoc]] LlavaForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6a4b96fd-2463-430b-8db4-9c78cc5ca3a7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron_gpt2", "title": "MegatronGPT2", "tokens": 1079, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6a4b96fd-2463-430b-8db4-9c78cc5ca3a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9ffcd9e7-19c3-42db-bec0-b9edc0fd0c63\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"51c15b44dd610b8061c3f2ec824126a3729e2738d6ba0df3d24c71a656bee71f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bfbb6914-c25c-4d4d-9aee-1eab95dafd0f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7583c9d3385b51d21aee21b4d8d703f3579209059f1a108a558d7b60900fec71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2851, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9ffcd9e7-19c3-42db-bec0-b9edc0fd0c63", "doc_id": "9ffcd9e7-19c3-42db-bec0-b9edc0fd0c63", "ref_doc_id": "9ffcd9e7-19c3-42db-bec0-b9edc0fd0c63", "chroma:document": "# MegatronGPT2\n\n## Overview\n\nThe MegatronGPT2 model was proposed in [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model\nParallelism](https://arxiv.org/abs/1909.08053) by Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley,\nJared Casper and Bryan Catanzaro.\n\nThe abstract from the paper is the following:\n\n*Recent work in language modeling demonstrates that training large transformer models advances the state of the art in\nNatural Language Processing applications. However, very large models can be quite difficult to train due to memory\nconstraints. In this work, we present our techniques for training very large transformer models and implement a simple,\nefficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our\napproach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model\nparallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We\nillustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain\n15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline\nthat sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance\nthe state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9\nbillion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in\nBERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we\nachieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA\naccuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy\nof 89.4%).*\n\nThis model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, it \ncontains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [GPT2-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_lm_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bfbb6914-c25c-4d4d-9aee-1eab95dafd0f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron_gpt2", "title": "MegatronGPT2", "tokens": 1079, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bfbb6914-c25c-4d4d-9aee-1eab95dafd0f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9ffcd9e7-19c3-42db-bec0-b9edc0fd0c63\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"51c15b44dd610b8061c3f2ec824126a3729e2738d6ba0df3d24c71a656bee71f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6a4b96fd-2463-430b-8db4-9c78cc5ca3a7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eed2c00cfaa89bdd5deaa61ed00e2b3a89e13a7c9465e74afa4160a6d1c93885\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1337, \"end_char_idx\": 3743, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9ffcd9e7-19c3-42db-bec0-b9edc0fd0c63", "doc_id": "9ffcd9e7-19c3-42db-bec0-b9edc0fd0c63", "ref_doc_id": "9ffcd9e7-19c3-42db-bec0-b9edc0fd0c63", "chroma:document": "To demonstrate that large language models can further advance\nthe state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9\nbillion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in\nBERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we\nachieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA\naccuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy\nof 89.4%).*\n\nThis model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, it \ncontains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [GPT2-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_lm_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).\n\nAlternatively, you can directly download the checkpoints using:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_lm_345m/versions/v0.0/zip -O\nmegatron_gpt2_345m_v0_0.zip\n```\n\nOnce you have obtained the checkpoint from NVIDIA GPU Cloud (NGC), you have to convert it to a format that will easily\nbe loaded by Hugging Face Transformers GPT2 implementation.\n\nThe following command allows you to do the conversion. We assume that the folder `models/megatron_gpt2` contains\n`megatron_gpt2_345m_v0_0.zip` and that the command is run from that folder:\n\n```bash\npython3 $PATH_TO_TRANSFORMERS/models/megatron_gpt2/convert_megatron_gpt2_checkpoint.py megatron_gpt2_345m_v0_0.zip\n```\n\n<Tip> \n\n MegatronGPT2 architecture is the same as OpenAI GPT-2 . Refer to [GPT-2 documentation](gpt2) for information on \n configuration classes and their parameters.  \n\n </Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f775f008-c254-4322-bbb2-1c32dd99f801
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nemotron", "title": "Nemotron", "tokens": 1387, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f775f008-c254-4322-bbb2-1c32dd99f801\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nemotron\", \"title\": \"Nemotron\", \"tokens\": 1387, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e109eae-5516-425c-84a3-9909ebdf64d0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nemotron\", \"title\": \"Nemotron\", \"tokens\": 1387, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4dff7633546eff20e4729970e23f840e7435b24abc649c3898c86956e9e3af60\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"18b92d4f-699b-4468-aef1-a7a43753a5c9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d0e33090a8d640ab0d4127fbfc157662ece3998dbfa17a1d70815dfd7ff16023\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3230, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e109eae-5516-425c-84a3-9909ebdf64d0", "doc_id": "4e109eae-5516-425c-84a3-9909ebdf64d0", "ref_doc_id": "4e109eae-5516-425c-84a3-9909ebdf64d0", "chroma:document": "# Nemotron\n\n## Nemotron\n\n### License\n\nThe use of this model is governed by the [NVIDIA AI Foundation Models Community License Agreement](https://developer.nvidia.com/downloads/nv-ai-foundation-models-license).\n\n### Description\n\nNemotron-4 is a family of enterprise ready generative text models compatible with [NVIDIA NeMo Framework](https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework/).\n\nNVIDIA NeMo is an end-to-end, cloud-native platform to build, customize, and deploy generative AI models anywhere. It includes training and inferencing frameworks, guardrailing toolkits, data curation tools, and pretrained models, offering enterprises an easy, cost-effective, and fast way to adopt generative AI. To get access to NeMo Framework, please sign up at [this link](https://developer.nvidia.com/nemo-framework/join).\n\n### References\n\n[Announcement Blog](https://developer.nvidia.com/blog/nvidia-ai-foundation-models-build-custom-enterprise-chatbots-and-co-pilots-with-production-ready-llms/)\n\n### Model Architecture\n\n**Architecture Type:** Transformer\n\n**Network Architecture:** Transformer Decoder (auto-regressive language model).\n\n## Minitron\n\n### Minitron 4B Base\n\nMinitron is a family of small language models (SLMs) obtained by pruning NVIDIA's [Nemotron-4 15B](https://arxiv.org/abs/2402.16819) model. We prune model embedding size, attention heads, and MLP intermediate dimension, following which, we perform continued training with distillation to arrive at the final models.\n\nDeriving the Minitron 8B and 4B models from the base 15B model using our approach requires up to **40x fewer training tokens** per model compared to training from scratch; this results in **compute cost savings of 1.8x** for training the full model family (15B, 8B, and 4B). Minitron models exhibit up to a 16% improvement in MMLU scores compared to training from scratch, perform comparably to other community models such as Mistral 7B, Gemma 7B and Llama-3 8B, and outperform state-of-the-art compression techniques from the literature. Please refer to our [arXiv paper](https://arxiv.org/abs/2407.14679) for more details.\n\nMinitron models are for research and development only.\n\n### HuggingFace Quickstart\n\nThe following code provides an example of how to load the Minitron-4B model and use it to perform text generation.\n\n```python\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\n# Load the tokenizer and model\nmodel_path = 'nvidia/Minitron-4B-Base'\ntokenizer  = AutoTokenizer.from_pretrained(model_path)\n\ndevice = 'cuda'\ndtype  = torch.bfloat16\nmodel  = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=dtype, device_map=device)\n\n# Prepare the input text\nprompt = 'Complete the paragraph: our solar system is'\ninputs = tokenizer.encode(prompt, return_tensors='pt').to(model.device)\n\n# Generate the output\noutputs = model.generate(inputs, max_length=20)\n\n# Decode and print the output\noutput_text = tokenizer.decode(outputs[0])\nprint(output_text)\n```\n\n### License\n\nMinitron is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).\n\n### Evaluation Results\n\n*5-shot performance."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 18b92d4f-699b-4468-aef1-a7a43753a5c9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nemotron", "title": "Nemotron", "tokens": 1387, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"18b92d4f-699b-4468-aef1-a7a43753a5c9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nemotron\", \"title\": \"Nemotron\", \"tokens\": 1387, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e109eae-5516-425c-84a3-9909ebdf64d0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nemotron\", \"title\": \"Nemotron\", \"tokens\": 1387, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4dff7633546eff20e4729970e23f840e7435b24abc649c3898c86956e9e3af60\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f775f008-c254-4322-bbb2-1c32dd99f801\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nemotron\", \"title\": \"Nemotron\", \"tokens\": 1387, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"99919fc1798b08fd3e82aca6187146fb0d5b044bafc11a0777c064b102c6bd11\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4da7fbdb-e754-4bea-8f1a-b6f037969135\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a5c605e9413fb30c9b6f52b6e505a99092c637315f56bd24cfde72345ef67d99\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1784, \"end_char_idx\": 4566, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e109eae-5516-425c-84a3-9909ebdf64d0", "doc_id": "4e109eae-5516-425c-84a3-9909ebdf64d0", "ref_doc_id": "4e109eae-5516-425c-84a3-9909ebdf64d0", "chroma:document": "Minitron models exhibit up to a 16% improvement in MMLU scores compared to training from scratch, perform comparably to other community models such as Mistral 7B, Gemma 7B and Llama-3 8B, and outperform state-of-the-art compression techniques from the literature. Please refer to our [arXiv paper](https://arxiv.org/abs/2407.14679) for more details.\n\nMinitron models are for research and development only.\n\n### HuggingFace Quickstart\n\nThe following code provides an example of how to load the Minitron-4B model and use it to perform text generation.\n\n```python\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\n# Load the tokenizer and model\nmodel_path = 'nvidia/Minitron-4B-Base'\ntokenizer  = AutoTokenizer.from_pretrained(model_path)\n\ndevice = 'cuda'\ndtype  = torch.bfloat16\nmodel  = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=dtype, device_map=device)\n\n# Prepare the input text\nprompt = 'Complete the paragraph: our solar system is'\ninputs = tokenizer.encode(prompt, return_tensors='pt').to(model.device)\n\n# Generate the output\noutputs = model.generate(inputs, max_length=20)\n\n# Decode and print the output\noutput_text = tokenizer.decode(outputs[0])\nprint(output_text)\n```\n\n### License\n\nMinitron is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).\n\n### Evaluation Results\n\n*5-shot performance.* Language Understanding evaluated using [Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300):\n\n| Average |\n| :---- |\n| 58.6 |\n\n*Zero-shot performance.* Evaluated using select datasets from the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) with additions:\n\n| HellaSwag | Winogrande | GSM8K| ARC-C | XLSum |\n| :------------- | :------------- | :------------- | :------------- | :------------- |\n| 75.0 | 74.0 | 24.1  | 50.9 | 29.5\n\n\n*Code generation performance*. Evaluated using [HumanEval](https://github.com/openai/human-eval):\n\n| p@1, 0-Shot |\n| :------------- |\n| 23.3 |\n\nPlease refer to our [paper](https://arxiv.org/abs/2407.14679) for the full set of results.\n\n### Citation\n\nIf you find our work helpful, please consider citing our paper:\n```\n@article{minitron2024,\n      title={Compact Language Models via Pruning and Knowledge Distillation},\n      author={Saurav Muralidharan and Sharath Turuvekere Sreenivas and Raviraj Joshi and Marcin Chochowski and Mostofa Patwary and Mohammad Shoeybi and Bryan Catanzaro and Jan Kautz and Pavlo Molchanov},\n      journal={arXiv preprint arXiv:2407.14679},\n      year={2024},\n      url={https://arxiv.org/abs/2407.14679},\n}\n```\n\n## NemotronConfig\n\n[[autodoc]] NemotronConfig\n\n\n## NemotronModel\n\n[[autodoc]] NemotronModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4da7fbdb-e754-4bea-8f1a-b6f037969135
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nemotron", "title": "Nemotron", "tokens": 1387, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4da7fbdb-e754-4bea-8f1a-b6f037969135\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nemotron\", \"title\": \"Nemotron\", \"tokens\": 1387, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e109eae-5516-425c-84a3-9909ebdf64d0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nemotron\", \"title\": \"Nemotron\", \"tokens\": 1387, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4dff7633546eff20e4729970e23f840e7435b24abc649c3898c86956e9e3af60\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"18b92d4f-699b-4468-aef1-a7a43753a5c9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nemotron\", \"title\": \"Nemotron\", \"tokens\": 1387, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"20343e1241be03de77265abd44cdd5a61bcc91ac95f56db43f83b309d88db04e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3186, \"end_char_idx\": 4921, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e109eae-5516-425c-84a3-9909ebdf64d0", "doc_id": "4e109eae-5516-425c-84a3-9909ebdf64d0", "ref_doc_id": "4e109eae-5516-425c-84a3-9909ebdf64d0", "chroma:document": "### Evaluation Results\n\n*5-shot performance.* Language Understanding evaluated using [Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300):\n\n| Average |\n| :---- |\n| 58.6 |\n\n*Zero-shot performance.* Evaluated using select datasets from the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) with additions:\n\n| HellaSwag | Winogrande | GSM8K| ARC-C | XLSum |\n| :------------- | :------------- | :------------- | :------------- | :------------- |\n| 75.0 | 74.0 | 24.1  | 50.9 | 29.5\n\n\n*Code generation performance*. Evaluated using [HumanEval](https://github.com/openai/human-eval):\n\n| p@1, 0-Shot |\n| :------------- |\n| 23.3 |\n\nPlease refer to our [paper](https://arxiv.org/abs/2407.14679) for the full set of results.\n\n### Citation\n\nIf you find our work helpful, please consider citing our paper:\n```\n@article{minitron2024,\n      title={Compact Language Models via Pruning and Knowledge Distillation},\n      author={Saurav Muralidharan and Sharath Turuvekere Sreenivas and Raviraj Joshi and Marcin Chochowski and Mostofa Patwary and Mohammad Shoeybi and Bryan Catanzaro and Jan Kautz and Pavlo Molchanov},\n      journal={arXiv preprint arXiv:2407.14679},\n      year={2024},\n      url={https://arxiv.org/abs/2407.14679},\n}\n```\n\n## NemotronConfig\n\n[[autodoc]] NemotronConfig\n\n\n## NemotronModel\n\n[[autodoc]] NemotronModel\n    - forward\n\n\n## NemotronForCausalLM\n\n[[autodoc]] NemotronForCausalLM\n    - forward\n\n## NemotronForSequenceClassification\n\n[[autodoc]] NemotronForSequenceClassification\n    - forward\n\n\n## NemotronForQuestionAnswering\n\n[[autodoc]] NemotronForQuestionAnswering\n    - forward\n\n\n## NemotronForTokenClassification\n\n[[autodoc]] NemotronForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 918c19dd-0f73-48b2-bb67-e2bd4db6603a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/opt", "title": "OPT", "tokens": 2021, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"918c19dd-0f73-48b2-bb67-e2bd4db6603a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4b789862-ddd0-4866-b61c-33203fd2172f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"09e3dc59e096fc84c942b2b30204d72154ca48ee72fc87f38a631eea8bb5909d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"66181d37-2c1a-4349-8240-b73dc25e5509\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f7f055c53d942dcf7bd6885f42a543f3bc115b623c1022562b2c34a63bb7c23e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3269, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "doc_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "ref_doc_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "chroma:document": "# OPT\n\n## Overview\n\nThe OPT model was proposed in [Open Pre-trained Transformer Language Models](https://arxiv.org/pdf/2205.01068) by Meta AI.\nOPT is a series of open-sourced large causal language models which perform similar in performance to GPT3.\n\nThe abstract from the paper is the following:\n\n*Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.*\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ), [Younes Belkada](https://huggingface.co/ybelkada), and [Patrick Von Platen](https://huggingface.co/patrickvonplaten).\nThe original code can be found [here](https://github.com/facebookresearch/metaseq).\n\nTips:\n- OPT has the same architecture as [`BartDecoder`].\n- Contrary to GPT2, OPT adds the EOS token `</s>` to the beginning of every prompt.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with OPT. If you're\ninterested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\" />\n\n- A notebook on [fine-tuning OPT with PEFT, bitsandbytes, and Transformers](https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing). \ud83c\udf0e\n- A blog post on [decoding strategies with OPT](https://huggingface.co/blog/introducing-csearch#62-example-two---opt).\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`OPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 66181d37-2c1a-4349-8240-b73dc25e5509
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/opt", "title": "OPT", "tokens": 2021, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"66181d37-2c1a-4349-8240-b73dc25e5509\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4b789862-ddd0-4866-b61c-33203fd2172f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"09e3dc59e096fc84c942b2b30204d72154ca48ee72fc87f38a631eea8bb5909d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"918c19dd-0f73-48b2-bb67-e2bd4db6603a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"82b6c07f450f4ac9c2489ef895528a624a57785be1cbedbeb24b0640fdce8de8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73be01c1-8fd8-4344-9958-5d53cd19ed19\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a6fa7f1420d2454099f517097181f8a15a52c5e9a7c24213f26a4bb980cf781\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1826, \"end_char_idx\": 4287, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "doc_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "ref_doc_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "chroma:document": "The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\" />\n\n- A notebook on [fine-tuning OPT with PEFT, bitsandbytes, and Transformers](https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing). \ud83c\udf0e\n- A blog post on [decoding strategies with OPT](https://huggingface.co/blog/introducing-csearch#62-example-two---opt).\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`OPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling).\n\n<PipelineTag pipeline=\"text-classification\" />\n\n- [Text classification task guide](sequence_classification.md)\n- [`OPTForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n\n<PipelineTag pipeline=\"question-answering\" />\n\n- [`OPTForQuestionAnswering`] is supported by this [question answering example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter\n  of the \ud83e\udd17 Hugging Face Course.\n\n\u26a1\ufe0f Inference\n\n- A blog post on [How \ud83e\udd17 Accelerate runs very large models thanks to PyTorch](https://huggingface.co/blog/accelerate-large-models) with OPT."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 73be01c1-8fd8-4344-9958-5d53cd19ed19
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/opt", "title": "OPT", "tokens": 2021, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"73be01c1-8fd8-4344-9958-5d53cd19ed19\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4b789862-ddd0-4866-b61c-33203fd2172f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"09e3dc59e096fc84c942b2b30204d72154ca48ee72fc87f38a631eea8bb5909d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"66181d37-2c1a-4349-8240-b73dc25e5509\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"13d4740ed28af8299325658fd3458e2fba134ef63a49af5790296bcf07c9f049\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"08e45946-cc42-478b-a1d1-a215b2c6928b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59cd8cf2338cf35f73d8c4972a32f31ed246242063a4619c0e1b9bb0d1d8283a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2754, \"end_char_idx\": 6804, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "doc_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "ref_doc_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "chroma:document": "- [`TFOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling).\n\n<PipelineTag pipeline=\"text-classification\" />\n\n- [Text classification task guide](sequence_classification.md)\n- [`OPTForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n\n<PipelineTag pipeline=\"question-answering\" />\n\n- [`OPTForQuestionAnswering`] is supported by this [question answering example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter\n  of the \ud83e\udd17 Hugging Face Course.\n\n\u26a1\ufe0f Inference\n\n- A blog post on [How \ud83e\udd17 Accelerate runs very large models thanks to PyTorch](https://huggingface.co/blog/accelerate-large-models) with OPT.\n\n\n## Combining OPT and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import OPTForCausalLM, GPT2Tokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = OPTForCausalLM.from_pretrained(\"facebook/opt-350m\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = GPT2Tokenizer.from_pretrained(\"facebook/opt-350m\")\n\n>>> prompt = (\"A chat between a curious human and the Statue of Liberty.\\n\\nHuman: What is your name?\\nStatue: I am the \"\n              \"Statue of Liberty.\\nHuman: Where do you live?\\nStatue: New York City.\\nHuman: How long have you lived \"\n              \"there?\")\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=30, do_sample=False)\n>>> tokenizer.batch_decode(generated_ids)[0]\n'</s>A chat between a curious human and the Statue of Liberty.\\n\\nHuman: What is your name?\\nStatue: I am the Statue of Liberty.\\nHuman: Where do you live?\\nStatue: New York City.\\nHuman: How long have you lived there?\\nStatue: I have lived here for about a year.\\nHuman: What is your favorite place to eat?\\nStatue: I love'\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `facebook/opt-2.7b` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.\n\n<div style=\"text-align: center\">\n<img src=\"https://user-images.githubusercontent.com/49240599/281101546-d2fca6d2-ee44-48f3-9534-ba8d5bee4531.png\">\n</div>\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `facebook/opt-350m` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.\n\n<div style=\"text-align: center\">\n<img src=\"https://user-images.githubusercontent.com/49240599/281101682-d1144e90-0dbc-46f4-8fc8-c6206cb793c9.png\">\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 08e45946-cc42-478b-a1d1-a215b2c6928b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/opt", "title": "OPT", "tokens": 2021, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"08e45946-cc42-478b-a1d1-a215b2c6928b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4b789862-ddd0-4866-b61c-33203fd2172f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"09e3dc59e096fc84c942b2b30204d72154ca48ee72fc87f38a631eea8bb5909d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73be01c1-8fd8-4344-9958-5d53cd19ed19\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9a255a14e08c540da2f2bf4a9beec1f52b596d79c1af97163bdbf21ff9cb009f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6808, \"end_char_idx\": 7439, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "doc_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "ref_doc_id": "4b789862-ddd0-4866-b61c-33203fd2172f", "chroma:document": "## OPTConfig\n\n[[autodoc]] OPTConfig\n\n<frameworkcontent>\n<pt>\n\n## OPTModel\n\n[[autodoc]] OPTModel\n    - forward\n\n## OPTForCausalLM\n\n[[autodoc]] OPTForCausalLM\n    - forward\n\n## OPTForSequenceClassification\n\n[[autodoc]] OPTForSequenceClassification\n    - forward\n\n## OPTForQuestionAnswering\n\n[[autodoc]] OPTForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFOPTModel\n\n[[autodoc]] TFOPTModel\n    - call\n\n## TFOPTForCausalLM\n\n[[autodoc]] TFOPTForCausalLM\n    - call\n\n</tf>\n<jax>\n\n## FlaxOPTModel\n\n[[autodoc]] FlaxOPTModel\n    - __call__\n\n## FlaxOPTForCausalLM\n\n[[autodoc]] FlaxOPTForCausalLM\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b85a52dc-5d7d-4fd8-8815-65ebd7b6d44c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5v1.1", "title": "T5v1.1", "tokens": 789, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b85a52dc-5d7d-4fd8-8815-65ebd7b6d44c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5v1.1\", \"title\": \"T5v1.1\", \"tokens\": 789, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ca1584f-efd5-4301-9e2d-08ffa5a4e70c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5v1.1\", \"title\": \"T5v1.1\", \"tokens\": 789, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15968be47294eb99869befa5f9933025e43f4feb1c2e5aa6829b41c83d3c5ed3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2292, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ca1584f-efd5-4301-9e2d-08ffa5a4e70c", "doc_id": "0ca1584f-efd5-4301-9e2d-08ffa5a4e70c", "ref_doc_id": "0ca1584f-efd5-4301-9e2d-08ffa5a4e70c", "chroma:document": "# T5v1.1\n\n## Overview\n\nT5v1.1 was released in the [google-research/text-to-text-transfer-transformer](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511)\nrepository by Colin Raffel et al. It's an improved version of the original T5 model.\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be\nfound [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511).\n\n## Usage tips\n\nOne can directly plug in the weights of T5v1.1 into a T5 model, like so:\n\n```python\n>>> from transformers import T5ForConditionalGeneration\n\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google/t5-v1_1-base\")\n```\n\nT5 Version 1.1 includes the following improvements compared to the original T5 model:\n\n- GEGLU activation in the feed-forward hidden layer, rather than ReLU. See [this paper](https://arxiv.org/abs/2002.05202).\n\n- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n\n- Pre-trained on C4 only without mixing in the downstream tasks.\n\n- No parameter sharing between the embedding and classifier layer.\n\n- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller\n  `num_heads` and `d_ff`.\n\nNote: T5 Version 1.1 was only pre-trained on [C4](https://huggingface.co/datasets/c4) excluding any supervised\ntraining. Therefore, this model has to be fine-tuned before it is usable on a downstream task, unlike the original T5\nmodel. Since t5v1.1 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\nGoogle has released the following variants:\n\n- [google/t5-v1_1-small](https://huggingface.co/google/t5-v1_1-small)\n\n- [google/t5-v1_1-base](https://huggingface.co/google/t5-v1_1-base)\n\n- [google/t5-v1_1-large](https://huggingface.co/google/t5-v1_1-large)\n\n- [google/t5-v1_1-xl](https://huggingface.co/google/t5-v1_1-xl)\n\n- [google/t5-v1_1-xxl](https://huggingface.co/google/t5-v1_1-xxl).\n\n\n<Tip>\n\nRefer to [T5's documentation page](t5) for all API reference, tips, code examples and notebooks.\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b57663da-451d-4a49-913e-81a9a848c93b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_mae", "title": "ViTMAE", "tokens": 1730, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b57663da-451d-4a49-913e-81a9a848c93b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c2a424c-5a11-4e65-84b7-55c6cb9adfa2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"82d9cbbbf1852ed8aea923719a26a2a7e19ca73e6530e1c77647d5dedc52b6cb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"84867df5-3589-4230-aa96-53427b019795\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fe91e45cfe245fd05511a1ff0059010a9da0ab510af224f47e0654713435778e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3151, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "doc_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "ref_doc_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "chroma:document": "# ViTMAE\n\n## Overview\n\nThe ViTMAE model was proposed in [Masked Autoencoders Are Scalable Vision Learners](https://arxiv.org/abs/2111.06377v2) by Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li,\nPiotr Doll\u00e1r, Ross Girshick. The paper shows that, by pre-training a Vision Transformer (ViT) to reconstruct pixel values for masked patches, one can get results after\nfine-tuning that outperform supervised pre-training.\n\nThe abstract from the paper is the following:\n\n*This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the\ninput image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates\nonly on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask\ntokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs\nenables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity\nmodels that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream\ntasks outperforms supervised pre-training and shows promising scaling behavior.*\n\n<img src=\"https://user-images.githubusercontent.com/11435359/146857310-f258c86c-fde6-48e8-9cee-badd2b21bd2c.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> MAE architecture. Taken from the <a href=\"https://arxiv.org/abs/2111.06377\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [sayakpaul](https://github.com/sayakpaul) and \n[ariG23498](https://github.com/ariG23498) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/mae). \n\n## Usage tips\n\n- MAE (masked auto encoding) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training objective is relatively simple:\nby masking a large portion (75%) of the image patches, the model must reconstruct raw pixel values. One can use [`ViTMAEForPreTraining`] for this purpose.\n- After pre-training, one \"throws away\" the decoder used to reconstruct pixels, and one uses the encoder for fine-tuning/linear probing. This means that after\nfine-tuning, one can directly plug in the weights into a [`ViTForImageClassification`].\n- One can use [`ViTImageProcessor`] to prepare images for the model. See the code examples for more info.\n- Note that the encoder of MAE is only used to encode the visual patches. The encoded patches are then concatenated with mask tokens, which the decoder (which also\nconsists of Transformer blocks) takes as input. Each mask token is a shared, learned vector that indicates the presence of a missing patch to be predicted."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 84867df5-3589-4230-aa96-53427b019795
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_mae", "title": "ViTMAE", "tokens": 1730, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"84867df5-3589-4230-aa96-53427b019795\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c2a424c-5a11-4e65-84b7-55c6cb9adfa2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"82d9cbbbf1852ed8aea923719a26a2a7e19ca73e6530e1c77647d5dedc52b6cb\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b57663da-451d-4a49-913e-81a9a848c93b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bc266869817cfe1993d6dbd1d835dc6e902bd3d36ed1e4264b41b5037eb181c8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"61c7ea1e-c29c-4117-b8b6-a44334bb3623\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9a142951efa4f276a55ebca0421f76529c6b01de5eb940cae1576029b2ff945d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1752, \"end_char_idx\": 4581, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "doc_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "ref_doc_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2111.06377\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [sayakpaul](https://github.com/sayakpaul) and \n[ariG23498](https://github.com/ariG23498) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/mae). \n\n## Usage tips\n\n- MAE (masked auto encoding) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training objective is relatively simple:\nby masking a large portion (75%) of the image patches, the model must reconstruct raw pixel values. One can use [`ViTMAEForPreTraining`] for this purpose.\n- After pre-training, one \"throws away\" the decoder used to reconstruct pixels, and one uses the encoder for fine-tuning/linear probing. This means that after\nfine-tuning, one can directly plug in the weights into a [`ViTForImageClassification`].\n- One can use [`ViTImageProcessor`] to prepare images for the model. See the code examples for more info.\n- Note that the encoder of MAE is only used to encode the visual patches. The encoded patches are then concatenated with mask tokens, which the decoder (which also\nconsists of Transformer blocks) takes as input. Each mask token is a shared, learned vector that indicates the presence of a missing patch to be predicted. Fixed\nsin/cos position embeddings are added both to the input of the encoder and the decoder.\n- For a visual understanding of how MAEs work you can check out this [post](https://keras.io/examples/vision/masked_image_modeling/).\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTMAEModel\nmodel = ViTMAEModel.from_pretrained(\"facebook/vit-mae-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-mae-base` model, we saw the following speedups during inference."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 61c7ea1e-c29c-4117-b8b6-a44334bb3623
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_mae", "title": "ViTMAE", "tokens": 1730, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"61c7ea1e-c29c-4117-b8b6-a44334bb3623\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c2a424c-5a11-4e65-84b7-55c6cb9adfa2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"82d9cbbbf1852ed8aea923719a26a2a7e19ca73e6530e1c77647d5dedc52b6cb\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"84867df5-3589-4230-aa96-53427b019795\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d3c4911e320c7183623ad1ee365284151b1c4e36bf0b707cd08ecb9e8ed60d27\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"99f1fed0-a2a0-4699-aaaa-8d8a03d074b9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"29256ad15e3d40cf9dc11e7b8fa1658e32bdfac4227953a4d06fb8d32da1a3db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3044, \"end_char_idx\": 6220, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "doc_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "ref_doc_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "chroma:document": "Each mask token is a shared, learned vector that indicates the presence of a missing patch to be predicted. Fixed\nsin/cos position embeddings are added both to the input of the encoder and the decoder.\n- For a visual understanding of how MAEs work you can check out this [post](https://keras.io/examples/vision/masked_image_modeling/).\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTMAEModel\nmodel = ViTMAEModel.from_pretrained(\"facebook/vit-mae-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-mae-base` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        11 |                                         6 |                      1.83 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViTMAE.\n\n- [`ViTMAEForPreTraining`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining), allowing you to pre-train the model from scratch/further pre-train the model on custom data.\n- A notebook that illustrates how to visualize reconstructed pixel values with [`ViTMAEForPreTraining`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/ViTMAE/ViT_MAE_visualization_demo.ipynb).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 99f1fed0-a2a0-4699-aaaa-8d8a03d074b9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_mae", "title": "ViTMAE", "tokens": 1730, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"99f1fed0-a2a0-4699-aaaa-8d8a03d074b9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c2a424c-5a11-4e65-84b7-55c6cb9adfa2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"82d9cbbbf1852ed8aea923719a26a2a7e19ca73e6530e1c77647d5dedc52b6cb\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"61c7ea1e-c29c-4117-b8b6-a44334bb3623\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"be4f1d20ad3509eff59e0171ac71fdc59c1ccd15948303c597a858548d5f5dab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4380, \"end_char_idx\": 6611, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "doc_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "ref_doc_id": "0c2a424c-5a11-4e65-84b7-55c6cb9adfa2", "chroma:document": "`torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-mae-base` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        11 |                                         6 |                      1.83 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViTMAE.\n\n- [`ViTMAEForPreTraining`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining), allowing you to pre-train the model from scratch/further pre-train the model on custom data.\n- A notebook that illustrates how to visualize reconstructed pixel values with [`ViTMAEForPreTraining`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/ViTMAE/ViT_MAE_visualization_demo.ipynb).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ViTMAEConfig\n\n[[autodoc]] ViTMAEConfig\n\n<frameworkcontent>\n<pt>\n\n## ViTMAEModel\n\n[[autodoc]] ViTMAEModel\n    - forward\n\n## ViTMAEForPreTraining\n\n[[autodoc]] transformers.ViTMAEForPreTraining\n    - forward\n\n</pt>\n<tf>\n\n## TFViTMAEModel\n\n[[autodoc]] TFViTMAEModel\n    - call\n\n## TFViTMAEForPreTraining\n\n[[autodoc]] transformers.TFViTMAEForPreTraining\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c01b4d84-344d-4871-bfbe-106b4b0558e1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ibert", "title": "I-BERT", "tokens": 822, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c01b4d84-344d-4871-bfbe-106b4b0558e1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ibert\", \"title\": \"I-BERT\", \"tokens\": 822, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ce6c7ccd-ee20-45e5-be62-3aff37287d62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ibert\", \"title\": \"I-BERT\", \"tokens\": 822, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"170f14f591ca018f081cb3c44352e01fa8e255de0b55798ce8418598c53c6982\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2741, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ce6c7ccd-ee20-45e5-be62-3aff37287d62", "doc_id": "ce6c7ccd-ee20-45e5-be62-3aff37287d62", "ref_doc_id": "ce6c7ccd-ee20-45e5-be62-3aff37287d62", "chroma:document": "# I-BERT\n\n## Overview\n\nThe I-BERT model was proposed in [I-BERT: Integer-only BERT Quantization](https://arxiv.org/abs/2101.01321) by\nSehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney and Kurt Keutzer. It's a quantized version of RoBERTa running\ninference up to four times faster.\n\nThe abstract from the paper is the following:\n\n*Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language\nProcessing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive for\nefficient inference at the edge, and even at the data center. While quantization can be a viable solution for this,\nprevious work on quantizing Transformer based models use floating-point arithmetic during inference, which cannot\nefficiently utilize integer-only logical units such as the recent Turing Tensor Cores, or traditional integer-only ARM\nprocessors. In this work, we propose I-BERT, a novel quantization scheme for Transformer based models that quantizes\nthe entire inference with integer-only arithmetic. Based on lightweight integer-only approximation methods for\nnonlinear operations, e.g., GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end integer-only BERT\ninference without any floating point calculation. We evaluate our approach on GLUE downstream tasks using\nRoBERTa-Base/Large. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to\nthe full-precision baseline. Furthermore, our preliminary implementation of I-BERT shows a speedup of 2.4 - 4.0x for\nINT8 inference on a T4 GPU system as compared to FP32 inference. The framework has been developed in PyTorch and has\nbeen open-sourced.*\n\nThis model was contributed by [kssteven](https://huggingface.co/kssteven). The original code can be found [here](https://github.com/kssteven418/I-BERT).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/masked_language_modeling)\n\n## IBertConfig\n\n[[autodoc]] IBertConfig\n\n## IBertModel\n\n[[autodoc]] IBertModel\n    - forward\n\n## IBertForMaskedLM\n\n[[autodoc]] IBertForMaskedLM\n    - forward\n\n## IBertForSequenceClassification\n\n[[autodoc]] IBertForSequenceClassification\n    - forward\n\n## IBertForMultipleChoice\n\n[[autodoc]] IBertForMultipleChoice\n    - forward\n\n## IBertForTokenClassification\n\n[[autodoc]] IBertForTokenClassification\n    - forward\n\n## IBertForQuestionAnswering\n\n[[autodoc]] IBertForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a1efd1a0-46bd-49cb-bb20-48666c71ca59
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/decision_transformer", "title": "Decision Transformer", "tokens": 552, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a1efd1a0-46bd-49cb-bb20-48666c71ca59\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/decision_transformer\", \"title\": \"Decision Transformer\", \"tokens\": 552, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bdec813-7aea-4ef6-8bda-b6409a447c50\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/decision_transformer\", \"title\": \"Decision Transformer\", \"tokens\": 552, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"310c5937c90c6fdf4e9457a440f343104364de8c9e9d6b89a89233b69e6b4bf9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1799, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bdec813-7aea-4ef6-8bda-b6409a447c50", "doc_id": "7bdec813-7aea-4ef6-8bda-b6409a447c50", "ref_doc_id": "7bdec813-7aea-4ef6-8bda-b6409a447c50", "chroma:document": "# Decision Transformer\n\n## Overview\n\nThe Decision Transformer model was proposed in [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://arxiv.org/abs/2106.01345)  \nby Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch.\n\nThe abstract from the paper is the following:\n\n*We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. \nThis allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances\n in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that \n casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or \n compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked \n Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our \n Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, \n Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on \n Atari, OpenAI Gym, and Key-to-Door tasks.*\n\nThis version of the model is for tasks where the state is a vector.\n\nThis model was contributed by [edbeeching](https://huggingface.co/edbeeching). The original code can be found [here](https://github.com/kzl/decision-transformer).\n\n## DecisionTransformerConfig\n\n[[autodoc]] DecisionTransformerConfig\n\n\n## DecisionTransformerGPT2Model\n\n[[autodoc]] DecisionTransformerGPT2Model\n    - forward\n\n## DecisionTransformerModel\n\n[[autodoc]] DecisionTransformerModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e6936b7d-4a4f-4327-b05b-e402135f79b0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pegasus", "title": "Pegasus", "tokens": 1626, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e6936b7d-4a4f-4327-b05b-e402135f79b0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e02eb03e-f552-4882-b8a5-b073ddd0d059\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5ff77b88d1acfb80bfc04093949ab311a43061ec7a8a660238dd7db384597ddd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e93ed55c-2262-4254-bfd7-dbeeaeeda9e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"138e36ec459e0fe7b6f521726551f9dbe65de5ba8be1f6d4b960f855b94e0c12\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3014, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e02eb03e-f552-4882-b8a5-b073ddd0d059", "doc_id": "e02eb03e-f552-4882-b8a5-b073ddd0d059", "ref_doc_id": "e02eb03e-f552-4882-b8a5-b073ddd0d059", "chroma:document": "# Pegasus\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=pegasus\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-pegasus-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/pegasus_paraphrase\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n\n## Overview\n\nThe Pegasus model was proposed in [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://arxiv.org/pdf/1912.08777.pdf) by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019.\n\nAccording to the abstract,\n\n- Pegasus' pretraining task is intentionally similar to summarization: important sentences are removed/masked from an\n  input document and are generated together as one output sequence from the remaining sentences, similar to an\n  extractive summary.\n- Pegasus achieves SOTA summarization performance on all 12 downstream tasks, as measured by ROUGE and human eval.\n\nThis model was contributed by [sshleifer](https://huggingface.co/sshleifer). The Authors' code can be found [here](https://github.com/google-research/pegasus).\n\n## Usage tips\n\n- Sequence-to-sequence model with the same encoder-decoder model architecture as BART. Pegasus is pre-trained jointly on two self-supervised objective functions: Masked Language Modeling (MLM) and a novel summarization specific pretraining objective, called Gap Sentence Generation (GSG).\n\n  * MLM: encoder input tokens are randomly replaced by a mask tokens and have to be predicted by the encoder (like in BERT)\n  * GSG: whole encoder input sentences are replaced by a second mask token and fed to the decoder, but which has a causal mask to hide the future words like a regular auto-regressive transformer decoder.\n\n- FP16 is not supported (help/ideas on this appreciated!).\n- The adafactor optimizer is recommended for pegasus fine-tuning.\n\n\n## Checkpoints\n\nAll the [checkpoints](https://huggingface.co/models?search=pegasus) are fine-tuned for summarization, besides\n*pegasus-large*, whence the other checkpoints are fine-tuned:\n\n- Each checkpoint is 2.2 GB on disk and 568M parameters.\n- FP16 is not supported (help/ideas on this appreciated!).\n- Summarizing xsum in fp32 takes about 400ms/sample, with default parameters on a v100 GPU.\n- Full replication results and correctly pre-processed data can be found in this [Issue](https://github.com/huggingface/transformers/issues/6844#issue-689259666).\n- [Distilled checkpoints](https://huggingface.co/models?search=distill-pegasus) are described in this [paper](https://arxiv.org/abs/2010.13002).\n\n## Implementation Notes\n\n- All models are transformer encoder-decoders with 16 layers in each component.\n- The implementation is completely inherited from [`BartForConditionalGeneration`]\n- Some key configuration differences:\n  - static, sinusoidal position embeddings\n  - the model starts generating with pad_token_id (which has 0 token_embedding) as the prefix."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e93ed55c-2262-4254-bfd7-dbeeaeeda9e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pegasus", "title": "Pegasus", "tokens": 1626, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e93ed55c-2262-4254-bfd7-dbeeaeeda9e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e02eb03e-f552-4882-b8a5-b073ddd0d059\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5ff77b88d1acfb80bfc04093949ab311a43061ec7a8a660238dd7db384597ddd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e6936b7d-4a4f-4327-b05b-e402135f79b0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"702c585bee6900605d026cca845434801627b4cd5efea23c6daaf33bea789327\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5bfdc6d5-e1c0-4fb5-b1d8-a30de1c4ec4f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7638b666072aeba6df2afb42d1648974cf72a28cc881b8e864f37a7423966c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1945, \"end_char_idx\": 5136, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e02eb03e-f552-4882-b8a5-b073ddd0d059", "doc_id": "e02eb03e-f552-4882-b8a5-b073ddd0d059", "ref_doc_id": "e02eb03e-f552-4882-b8a5-b073ddd0d059", "chroma:document": "## Checkpoints\n\nAll the [checkpoints](https://huggingface.co/models?search=pegasus) are fine-tuned for summarization, besides\n*pegasus-large*, whence the other checkpoints are fine-tuned:\n\n- Each checkpoint is 2.2 GB on disk and 568M parameters.\n- FP16 is not supported (help/ideas on this appreciated!).\n- Summarizing xsum in fp32 takes about 400ms/sample, with default parameters on a v100 GPU.\n- Full replication results and correctly pre-processed data can be found in this [Issue](https://github.com/huggingface/transformers/issues/6844#issue-689259666).\n- [Distilled checkpoints](https://huggingface.co/models?search=distill-pegasus) are described in this [paper](https://arxiv.org/abs/2010.13002).\n\n## Implementation Notes\n\n- All models are transformer encoder-decoders with 16 layers in each component.\n- The implementation is completely inherited from [`BartForConditionalGeneration`]\n- Some key configuration differences:\n  - static, sinusoidal position embeddings\n  - the model starts generating with pad_token_id (which has 0 token_embedding) as the prefix.\n  - more beams are used (`num_beams=8`)\n- All pretrained pegasus checkpoints are the same besides three attributes: `tokenizer.model_max_length` (maximum\n  input size), `max_length` (the maximum number of tokens to generate) and `length_penalty`.\n- The code to convert checkpoints trained in the author's [repo](https://github.com/google-research/pegasus) can be\n  found in `convert_pegasus_tf_to_pytorch.py`.\n\n## Usage Example\n\n```python\n>>> from transformers import PegasusForConditionalGeneration, PegasusTokenizer\n>>> import torch\n\n>>> src_text = [\n...     \"\"\" PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.\"\"\"\n... ]\n\n... model_name = \"google/pegasus-xsum\"\n... device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n... tokenizer = PegasusTokenizer.from_pretrained(model_name)\n... model = PegasusForConditionalGeneration.from_pretrained(model_name).to(device)\n... batch = tokenizer(src_text, truncation=True, padding=\"longest\", return_tensors=\"pt\").to(device)\n... translated = model.generate(**batch)\n... tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)\n... assert (\n...     tgt_text[0]\n...     == \"California's largest electricity provider has turned off power to hundreds of thousands of customers.\"\n... )\n```\n\n## Resources\n\n- [Script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/seq2seq-distillation/finetune_pegasus_xsum.sh) to fine-tune pegasus\n  on the XSUM dataset. Data download instructions at [examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md).\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## PegasusConfig\n\n[[autodoc]] PegasusConfig\n\n## PegasusTokenizer\n\nwarning: `add_tokens` does not work at the moment."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5bfdc6d5-e1c0-4fb5-b1d8-a30de1c4ec4f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pegasus", "title": "Pegasus", "tokens": 1626, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5bfdc6d5-e1c0-4fb5-b1d8-a30de1c4ec4f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e02eb03e-f552-4882-b8a5-b073ddd0d059\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5ff77b88d1acfb80bfc04093949ab311a43061ec7a8a660238dd7db384597ddd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e93ed55c-2262-4254-bfd7-dbeeaeeda9e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7eff0e9cc8ce2a5080cbbb5cfc1d11573d75f835ee3b77d0063562678d9b0d5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3426, \"end_char_idx\": 5900, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e02eb03e-f552-4882-b8a5-b073ddd0d059", "doc_id": "e02eb03e-f552-4882-b8a5-b073ddd0d059", "ref_doc_id": "e02eb03e-f552-4882-b8a5-b073ddd0d059", "chroma:document": "## Usage Example\n\n```python\n>>> from transformers import PegasusForConditionalGeneration, PegasusTokenizer\n>>> import torch\n\n>>> src_text = [\n...     \"\"\" PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.\"\"\"\n... ]\n\n... model_name = \"google/pegasus-xsum\"\n... device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n... tokenizer = PegasusTokenizer.from_pretrained(model_name)\n... model = PegasusForConditionalGeneration.from_pretrained(model_name).to(device)\n... batch = tokenizer(src_text, truncation=True, padding=\"longest\", return_tensors=\"pt\").to(device)\n... translated = model.generate(**batch)\n... tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)\n... assert (\n...     tgt_text[0]\n...     == \"California's largest electricity provider has turned off power to hundreds of thousands of customers.\"\n... )\n```\n\n## Resources\n\n- [Script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/seq2seq-distillation/finetune_pegasus_xsum.sh) to fine-tune pegasus\n  on the XSUM dataset. Data download instructions at [examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md).\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## PegasusConfig\n\n[[autodoc]] PegasusConfig\n\n## PegasusTokenizer\n\nwarning: `add_tokens` does not work at the moment.\n\n[[autodoc]] PegasusTokenizer\n\n## PegasusTokenizerFast\n\n[[autodoc]] PegasusTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## PegasusModel\n\n[[autodoc]] PegasusModel\n    - forward\n\n## PegasusForConditionalGeneration\n\n[[autodoc]] PegasusForConditionalGeneration\n    - forward\n\n## PegasusForCausalLM\n\n[[autodoc]] PegasusForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFPegasusModel\n\n[[autodoc]] TFPegasusModel\n    - call\n\n## TFPegasusForConditionalGeneration\n\n[[autodoc]] TFPegasusForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxPegasusModel\n\n[[autodoc]] FlaxPegasusModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxPegasusForConditionalGeneration\n\n[[autodoc]] FlaxPegasusForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3f811ce3-bf59-4127-ac75-114e1147cb57
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/poolformer", "title": "PoolFormer", "tokens": 1298, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3f811ce3-bf59-4127-ac75-114e1147cb57\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"50474c19-2987-4680-a756-d5a40b37d918\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"25141c0af9221c820f9da4694650475d9889b2dcde2be69afe1d22c08c62b1f4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ed54b92f-f89d-4e49-b0f9-924449c3cae1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b5c9d53c9b53be72d77863736dc2b0db5705437b1f7c26c17f3ff9718e0c046\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2901, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "50474c19-2987-4680-a756-d5a40b37d918", "doc_id": "50474c19-2987-4680-a756-d5a40b37d918", "ref_doc_id": "50474c19-2987-4680-a756-d5a40b37d918", "chroma:document": "# PoolFormer\n\n## Overview\n\nThe PoolFormer model was proposed in [MetaFormer is Actually What You Need for Vision](https://arxiv.org/abs/2111.11418)  by Sea AI Labs. Instead of designing complicated token mixer to achieve SOTA performance, the target of this work is to demonstrate the competence of transformer models largely stem from the general architecture MetaFormer.\n\nThe abstract from the paper is the following:\n\n*Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in transformers with an embarrassingly simple spatial pooling operator to conduct only the most basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned vision transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 48%/60% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of \"MetaFormer\", a general architecture abstracted from transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design.*\n\nThe figure below illustrates the architecture of PoolFormer. Taken from the [original paper](https://arxiv.org/abs/2111.11418).\n\n<img width=\"600\" src=\"https://user-images.githubusercontent.com/15921929/142746124-1ab7635d-2536-4a0e-ad43-b4fe2c5a525d.png\"/>\n\nThis model was contributed by [heytanay](https://huggingface.co/heytanay). The original code can be found [here](https://github.com/sail-sg/poolformer).\n\n## Usage tips\n\n- PoolFormer has a hierarchical architecture, where instead of Attention, a simple Average Pooling layer is present. All checkpoints of the model can be found on the [hub](https://huggingface.co/models?other=poolformer).\n- One can use [`PoolFormerImageProcessor`] to prepare images for the model.\n- As most models, PoolFormer comes in different sizes, the details of which can be found in the table below."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ed54b92f-f89d-4e49-b0f9-924449c3cae1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/poolformer", "title": "PoolFormer", "tokens": 1298, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ed54b92f-f89d-4e49-b0f9-924449c3cae1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"50474c19-2987-4680-a756-d5a40b37d918\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"25141c0af9221c820f9da4694650475d9889b2dcde2be69afe1d22c08c62b1f4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3f811ce3-bf59-4127-ac75-114e1147cb57\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d823d3136501e858908a6634c2e89291a49fe5a846848fdad549f56c94930039\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"91bdf779-f905-42c8-b3dd-b37c9c1318bf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a2ddb5b7b3030213d9b235a50b519c68ffc8d595070ea28c851d660df2096a43\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1237, \"end_char_idx\": 4083, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "50474c19-2987-4680-a756-d5a40b37d918", "doc_id": "50474c19-2987-4680-a756-d5a40b37d918", "ref_doc_id": "50474c19-2987-4680-a756-d5a40b37d918", "chroma:document": "For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned vision transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 48%/60% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of \"MetaFormer\", a general architecture abstracted from transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design.*\n\nThe figure below illustrates the architecture of PoolFormer. Taken from the [original paper](https://arxiv.org/abs/2111.11418).\n\n<img width=\"600\" src=\"https://user-images.githubusercontent.com/15921929/142746124-1ab7635d-2536-4a0e-ad43-b4fe2c5a525d.png\"/>\n\nThis model was contributed by [heytanay](https://huggingface.co/heytanay). The original code can be found [here](https://github.com/sail-sg/poolformer).\n\n## Usage tips\n\n- PoolFormer has a hierarchical architecture, where instead of Attention, a simple Average Pooling layer is present. All checkpoints of the model can be found on the [hub](https://huggingface.co/models?other=poolformer).\n- One can use [`PoolFormerImageProcessor`] to prepare images for the model.\n- As most models, PoolFormer comes in different sizes, the details of which can be found in the table below.\n\n| **Model variant** | **Depths**    | **Hidden sizes**    | **Params (M)** | **ImageNet-1k Top 1** |\n| :---------------: | ------------- | ------------------- | :------------: | :-------------------: |\n| s12               | [2, 2, 6, 2]  | [64, 128, 320, 512] | 12             | 77.2                  |\n| s24               | [4, 4, 12, 4] | [64, 128, 320, 512] | 21             | 80.3                  |\n| s36               | [6, 6, 18, 6] | [64, 128, 320, 512] | 31             | 81.4                  |\n| m36               | [6, 6, 18, 6] | [96, 192, 384, 768] | 56             | 82.1                  |\n| m48               | [8, 8, 24, 8] | [96, 192, 384, 768] | 73             | 82.5                  |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with PoolFormer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`PoolFormerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 91bdf779-f905-42c8-b3dd-b37c9c1318bf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/poolformer", "title": "PoolFormer", "tokens": 1298, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"91bdf779-f905-42c8-b3dd-b37c9c1318bf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"50474c19-2987-4680-a756-d5a40b37d918\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"25141c0af9221c820f9da4694650475d9889b2dcde2be69afe1d22c08c62b1f4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ed54b92f-f89d-4e49-b0f9-924449c3cae1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0f73e2a827e9383d75ba23ed38576e84cd0a47bcac7236f9348c8af97fcfb4f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2793, \"end_char_idx\": 4769, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "50474c19-2987-4680-a756-d5a40b37d918", "doc_id": "50474c19-2987-4680-a756-d5a40b37d918", "ref_doc_id": "50474c19-2987-4680-a756-d5a40b37d918", "chroma:document": "- As most models, PoolFormer comes in different sizes, the details of which can be found in the table below.\n\n| **Model variant** | **Depths**    | **Hidden sizes**    | **Params (M)** | **ImageNet-1k Top 1** |\n| :---------------: | ------------- | ------------------- | :------------: | :-------------------: |\n| s12               | [2, 2, 6, 2]  | [64, 128, 320, 512] | 12             | 77.2                  |\n| s24               | [4, 4, 12, 4] | [64, 128, 320, 512] | 21             | 80.3                  |\n| s36               | [6, 6, 18, 6] | [64, 128, 320, 512] | 31             | 81.4                  |\n| m36               | [6, 6, 18, 6] | [96, 192, 384, 768] | 56             | 82.1                  |\n| m48               | [8, 8, 24, 8] | [96, 192, 384, 768] | 73             | 82.5                  |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with PoolFormer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`PoolFormerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## PoolFormerConfig\n\n[[autodoc]] PoolFormerConfig\n\n## PoolFormerFeatureExtractor\n\n[[autodoc]] PoolFormerFeatureExtractor\n    - __call__\n\n## PoolFormerImageProcessor\n\n[[autodoc]] PoolFormerImageProcessor\n    - preprocess\n\n## PoolFormerModel\n\n[[autodoc]] PoolFormerModel\n    - forward\n\n## PoolFormerForImageClassification\n\n[[autodoc]] PoolFormerForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2ea6e085-4362-46fb-afe1-f48e157d1e67
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon", "title": "Chameleon", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2ea6e085-4362-46fb-afe1-f48e157d1e67\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f861900-5d30-41d4-8112-eaf1f192faeb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c26d3c1d935425ebd8716acd876d96838f1ab9836c4da4c20d6754bb98051d55\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7d34d0c4-1e8f-4374-8020-57703a0be373\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59230c6179a8780bd62a1ba21ec0d14d2352aef58d3bd044d41cf5eddebc1b6a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3466, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "ref_doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "chroma:document": "# Chameleon\n\n## Overview\n\nThe Chameleon model was proposed in [Chameleon: Mixed-Modal Early-Fusion Foundation Models\n](https://arxiv.org/abs/2405.09818v1) by META AI Chameleon Team. Chameleon is a Vision-Language Model that use vector quantization to tokenize images which enables the model to generate multimodal output. The model takes images and texts as input, including an interleaved format, and generates textual response. Image generation module is not released yet. \n\n\nThe abstract from the paper is the following:\n\n*We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training\napproach from inception, an alignment recipe, and an architectural parameterization tailored for the\nearly-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range\nof tasks, including visual question answering, image captioning, text generation, image generation, and\nlong-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including\nstate-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while\nbeing competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image\ngeneration, all in a single model. It also matches or exceeds the performance of much larger models,\nincluding Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal\ngeneration evaluation, where either the prompt or outputs contain mixed sequences of both images and\ntext. Chameleon marks a significant step forward in unified modeling of full multimodal documents*\n\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/chameleon_arch.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Chameleon incorporates a vector quantizer module to transform images into discrete tokens. That also enables image generation using an auto-regressive transformer. Taken from the <a href=\"https://arxiv.org/abs/2405.09818v1\">original paper.</a> </small>\n\nThis model was contributed by [joaogante](https://huggingface.co/joaogante) and [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).\nThe original code can be found [here](https://github.com/facebookresearch/chameleon).\n\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to set `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note that Chameleon was tuned for safety alignment. If the model is refusing to answer, consider asking a more concrete question, instead of an open question.\n\n- Chameleon generates in chat format which means that the generated text will always be the \"assistant's turn\". You can enable a text completion generation by passing `return_for_text_completion=True` when calling the processor.\n\n> [!NOTE]\n> Chameleon implementation in Transformers uses a special image token to indicate where to merge image embeddings. For special image token we didn't add a new one but used one of the reserved tokens: `<reserved08707>`. You have to add `<image>` to your prompt in the place where the image should be embedded for correct generation.\n\n## Usage example\n\n### Single image inference\n\nChameleon is a gated model so make sure to have access and login to Hugging Face Hub using a token."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7d34d0c4-1e8f-4374-8020-57703a0be373
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon", "title": "Chameleon", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7d34d0c4-1e8f-4374-8020-57703a0be373\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f861900-5d30-41d4-8112-eaf1f192faeb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c26d3c1d935425ebd8716acd876d96838f1ab9836c4da4c20d6754bb98051d55\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2ea6e085-4362-46fb-afe1-f48e157d1e67\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7c87828a2303b18e48438016d266e1f2ca3eaf55aaaf0f19a1246dd76e900c29\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4780caaf-9051-49ee-b804-818c2f813bcd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5fbf2ffc5190d38f41ae62d879e739a118ad5f5e171dfe991c771353321b7129\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1713, \"end_char_idx\": 4509, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "ref_doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "chroma:document": "<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/chameleon_arch.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Chameleon incorporates a vector quantizer module to transform images into discrete tokens. That also enables image generation using an auto-regressive transformer. Taken from the <a href=\"https://arxiv.org/abs/2405.09818v1\">original paper.</a> </small>\n\nThis model was contributed by [joaogante](https://huggingface.co/joaogante) and [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).\nThe original code can be found [here](https://github.com/facebookresearch/chameleon).\n\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to set `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note that Chameleon was tuned for safety alignment. If the model is refusing to answer, consider asking a more concrete question, instead of an open question.\n\n- Chameleon generates in chat format which means that the generated text will always be the \"assistant's turn\". You can enable a text completion generation by passing `return_for_text_completion=True` when calling the processor.\n\n> [!NOTE]\n> Chameleon implementation in Transformers uses a special image token to indicate where to merge image embeddings. For special image token we didn't add a new one but used one of the reserved tokens: `<reserved08707>`. You have to add `<image>` to your prompt in the place where the image should be embedded for correct generation.\n\n## Usage example\n\n### Single image inference\n\nChameleon is a gated model so make sure to have access and login to Hugging Face Hub using a token. \nHere's how to load the model and perform inference in half-precision (`torch.bfloat16`):\n\n```python\nfrom transformers import ChameleonProcessor, ChameleonForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = ChameleonProcessor.from_pretrained(\"facebook/chameleon-7b\")\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", torch_dtype=torch.bfloat16, device_map=\"cuda\")\n\n# prepare image and text prompt\nurl = 'http://images.cocodataset.org/val2017/000000039769.jpg'\nimage = Image.open(requests.get(url, stream=True).raw)\nprompt = \"What do you see in this image?<image>\"\n\ninputs = processor(prompt, image, return_tensors=\"pt\").to(model.device)\n\n# autoregressively complete prompt\noutput = model.generate(**inputs, max_new_tokens=50)\nprint(processor.decode(output[0], skip_special_tokens=True))\n```\n\n### Multi image inference\n\nChameleon can perform inference with multiple images as input, where images either belong to the same prompt or different prompts (in batched inference)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4780caaf-9051-49ee-b804-818c2f813bcd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon", "title": "Chameleon", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4780caaf-9051-49ee-b804-818c2f813bcd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f861900-5d30-41d4-8112-eaf1f192faeb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c26d3c1d935425ebd8716acd876d96838f1ab9836c4da4c20d6754bb98051d55\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7d34d0c4-1e8f-4374-8020-57703a0be373\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4be0bfb3d282938186f5b2871e7358734fe9783dcb95e3d4ad1b69cbdf35bc5d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25600080-cf28-42a8-866d-9ba44f2376b6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b8bb1063caa6060016c82dcad30792d42959a1ab82f7b6f8388729367804579c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2748, \"end_char_idx\": 5512, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "ref_doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "chroma:document": "- Chameleon generates in chat format which means that the generated text will always be the \"assistant's turn\". You can enable a text completion generation by passing `return_for_text_completion=True` when calling the processor.\n\n> [!NOTE]\n> Chameleon implementation in Transformers uses a special image token to indicate where to merge image embeddings. For special image token we didn't add a new one but used one of the reserved tokens: `<reserved08707>`. You have to add `<image>` to your prompt in the place where the image should be embedded for correct generation.\n\n## Usage example\n\n### Single image inference\n\nChameleon is a gated model so make sure to have access and login to Hugging Face Hub using a token. \nHere's how to load the model and perform inference in half-precision (`torch.bfloat16`):\n\n```python\nfrom transformers import ChameleonProcessor, ChameleonForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = ChameleonProcessor.from_pretrained(\"facebook/chameleon-7b\")\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", torch_dtype=torch.bfloat16, device_map=\"cuda\")\n\n# prepare image and text prompt\nurl = 'http://images.cocodataset.org/val2017/000000039769.jpg'\nimage = Image.open(requests.get(url, stream=True).raw)\nprompt = \"What do you see in this image?<image>\"\n\ninputs = processor(prompt, image, return_tensors=\"pt\").to(model.device)\n\n# autoregressively complete prompt\noutput = model.generate(**inputs, max_new_tokens=50)\nprint(processor.decode(output[0], skip_special_tokens=True))\n```\n\n### Multi image inference\n\nChameleon can perform inference with multiple images as input, where images either belong to the same prompt or different prompts (in batched inference). Here is how you can do it:\n\n```python\nfrom transformers import ChameleonProcessor, ChameleonForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = ChameleonProcessor.from_pretrained(\"facebook/chameleon-7b\")\n\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", torch_dtype=torch.bfloat16, device_map=\"cuda\")\n\n# Get three different images\nurl = \"https://www.ilankelman.org/stopsigns/australia.jpg\"\nimage_stop = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage_cats = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\nimage_snowman = Image.open(requests.get(url, stream=True).raw)\n\n# Prepare a batched prompt, where the first one is a multi-image prompt and the second is not\nprompts = [\n    \"What do these images have in common?<image><image>\",\n    \"<image>What is shown in this image?\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 25600080-cf28-42a8-866d-9ba44f2376b6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon", "title": "Chameleon", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"25600080-cf28-42a8-866d-9ba44f2376b6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f861900-5d30-41d4-8112-eaf1f192faeb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c26d3c1d935425ebd8716acd876d96838f1ab9836c4da4c20d6754bb98051d55\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4780caaf-9051-49ee-b804-818c2f813bcd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d21fda4e0bbf094b94bf4dc96e3705e729227a88a46e9a8281de69c7e708dc2f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"070d2b8f-19fa-449a-9f19-e701e1bb9454\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e45a39c6ae05fa96e40ce5fc43d861105866025ae9f909c2ba7037653e8aafb0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4510, \"end_char_idx\": 7370, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "ref_doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "chroma:document": "Here is how you can do it:\n\n```python\nfrom transformers import ChameleonProcessor, ChameleonForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = ChameleonProcessor.from_pretrained(\"facebook/chameleon-7b\")\n\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", torch_dtype=torch.bfloat16, device_map=\"cuda\")\n\n# Get three different images\nurl = \"https://www.ilankelman.org/stopsigns/australia.jpg\"\nimage_stop = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage_cats = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\nimage_snowman = Image.open(requests.get(url, stream=True).raw)\n\n# Prepare a batched prompt, where the first one is a multi-image prompt and the second is not\nprompts = [\n    \"What do these images have in common?<image><image>\",\n    \"<image>What is shown in this image?\"\n]\n\n# We can simply feed images in the order they have to be used in the text prompt\n# Each \"<image>\" token uses one image leaving the next for the subsequent \"<image>\" tokens\ninputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors=\"pt\").to(device=\"cuda\", dtype=torch.bfloat16)\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_new_tokens=50)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes\n\nThe model can be loaded in 8 or 4 bits, greatly reducing the memory requirements while maintaining the performance of the original model. First make sure to install bitsandbytes, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:\n\n```python\nfrom transformers import ChameleonForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.bfloat16,\n)\n\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", quantization_config=quantization_config, device_map=\"cuda\")\n```\n\n### Use Flash-Attention 2 and SDPA to further speed-up generation\n\nThe models supports both, Flash-Attention 2 and PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) which can be enables for optimization. SDPA is the default options when you load the model, If you want to switch for Flash Attention 2, first make sure to install flash-attn. Refer to the [original repository](https://github.com/Dao-AILab/flash-attention) regarding that package installation."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 070d2b8f-19fa-449a-9f19-e701e1bb9454
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon", "title": "Chameleon", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"070d2b8f-19fa-449a-9f19-e701e1bb9454\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f861900-5d30-41d4-8112-eaf1f192faeb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c26d3c1d935425ebd8716acd876d96838f1ab9836c4da4c20d6754bb98051d55\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25600080-cf28-42a8-866d-9ba44f2376b6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon\", \"title\": \"Chameleon\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"32b762ef6c09e2a18860f36efed81a4b0b29d53e7992ec5440aff33b8055fe1b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6217, \"end_char_idx\": 8171, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "ref_doc_id": "8f861900-5d30-41d4-8112-eaf1f192faeb", "chroma:document": "First make sure to install bitsandbytes, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:\n\n```python\nfrom transformers import ChameleonForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.bfloat16,\n)\n\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", quantization_config=quantization_config, device_map=\"cuda\")\n```\n\n### Use Flash-Attention 2 and SDPA to further speed-up generation\n\nThe models supports both, Flash-Attention 2 and PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) which can be enables for optimization. SDPA is the default options when you load the model, If you want to switch for Flash Attention 2, first make sure to install flash-attn. Refer to the [original repository](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with:\n\n```python\nfrom transformers import ChameleonForConditionalGeneration\n\nmodel_id = \"facebook/chameleon-7b\"\nmodel = ChameleonForConditionalGeneration.from_pretrained(\n    model_id, \n    torch_dtype=torch.bfloat16, \n    low_cpu_mem_usage=True,\n    attn_implementation=\"flash_attention_2\"\n).to(0)\n```\n\n## ChameleonConfig\n\n[[autodoc]] ChameleonConfig\n\n## ChameleonVQVAEConfig\n\n[[autodoc]] ChameleonVQVAEConfig\n\n## ChameleonProcessor\n\n[[autodoc]] ChameleonProcessor\n\n## ChameleonImageProcessor\n\n[[autodoc]] ChameleonImageProcessor\n    - preprocess\n\n## ChameleonVQVAE\n\n[[autodoc]] ChameleonVQVAE\n    - forward\n\n## ChameleonModel\n\n[[autodoc]] ChameleonModel\n    - forward\n\n## ChameleonForConditionalGeneration\n\n[[autodoc]] ChameleonForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3428aa38-e2df-4ee0-b9dd-82f3020feafc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yoso", "title": "YOSO", "tokens": 1066, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3428aa38-e2df-4ee0-b9dd-82f3020feafc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c90ace85-c7f5-488c-97b3-85a19e7f7286\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10df9218541cb870446642c7ae948733653d6280b9818b27c9c86031a22b5a7c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a014a9a4-cbf0-478d-aeab-e7bd76ad217f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4cdce73741b35c392b7e3d6fbab67bfbb79d635c475503899c19d9ab5106faf0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2986, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c90ace85-c7f5-488c-97b3-85a19e7f7286", "doc_id": "c90ace85-c7f5-488c-97b3-85a19e7f7286", "ref_doc_id": "c90ace85-c7f5-488c-97b3-85a19e7f7286", "chroma:document": "# YOSO\n\n## Overview\n\nThe YOSO model was proposed in [You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling](https://arxiv.org/abs/2111.09714)  \nby Zhanpeng Zeng, Yunyang Xiong, Sathya N. Ravi, Shailesh Acharya, Glenn Fung, Vikas Singh. YOSO approximates standard softmax self-attention\nvia a Bernoulli sampling scheme based on Locality Sensitive Hashing (LSH). In principle, all the Bernoulli random variables can be sampled with\na single hash. \n\nThe abstract from the paper is the following:\n\n*Transformer-based models are widely used in natural language processing (NLP). Central to the transformer model is \nthe self-attention mechanism, which captures the interactions of token pairs in the input sequences and depends quadratically \non the sequence length. Training such models on longer sequences is expensive. In this paper, we show that a Bernoulli sampling \nattention mechanism based on Locality Sensitive Hashing (LSH), decreases the quadratic complexity of such models to linear. \nWe bypass the quadratic cost by considering self-attention as a sum of individual tokens associated with Bernoulli random \nvariables that can, in principle, be sampled at once by a single hash (although in practice, this number may be a small constant). \nThis leads to an efficient sampling scheme to estimate self-attention which relies on specific modifications of \nLSH (to enable deployment on GPU architectures). We evaluate our algorithm on the GLUE benchmark with standard 512 sequence \nlength where we see favorable performance relative to a standard pretrained Transformer. On the Long Range Arena (LRA) benchmark, \nfor evaluating performance on long sequences, our method achieves results consistent with softmax self-attention but with sizable \nspeed-ups and memory savings and often outperforms other efficient self-attention methods. Our code is available at this https URL*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The original code can be found [here](https://github.com/mlpen/YOSO).\n\n## Usage tips\n\n- The YOSO attention algorithm is implemented through custom CUDA kernels, functions written in CUDA C++ that can be executed multiple times\nin parallel on a GPU.\n- The kernels provide a `fast_hash` function, which approximates the random projections of the queries and keys using the Fast Hadamard Transform. Using these\nhash codes, the `lsh_cumulation` function approximates self-attention via LSH-based Bernoulli sampling.\n- To use the custom kernels, the user should set `config.use_expectation = False`. To ensure that the kernels are compiled successfully, \nthe user must install the correct version of PyTorch and cudatoolkit. By default, `config.use_expectation = True`, which uses YOSO-E and \ndoes not require compiling CUDA kernels.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/yoso_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> YOSO Attention Algorithm."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a014a9a4-cbf0-478d-aeab-e7bd76ad217f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yoso", "title": "YOSO", "tokens": 1066, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a014a9a4-cbf0-478d-aeab-e7bd76ad217f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c90ace85-c7f5-488c-97b3-85a19e7f7286\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10df9218541cb870446642c7ae948733653d6280b9818b27c9c86031a22b5a7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3428aa38-e2df-4ee0-b9dd-82f3020feafc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2bafcbc49ccaac307b8bdaf627e823cf2627f432ca2c081cb55efdc65595f215\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1274, \"end_char_idx\": 3903, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c90ace85-c7f5-488c-97b3-85a19e7f7286", "doc_id": "c90ace85-c7f5-488c-97b3-85a19e7f7286", "ref_doc_id": "c90ace85-c7f5-488c-97b3-85a19e7f7286", "chroma:document": "This leads to an efficient sampling scheme to estimate self-attention which relies on specific modifications of \nLSH (to enable deployment on GPU architectures). We evaluate our algorithm on the GLUE benchmark with standard 512 sequence \nlength where we see favorable performance relative to a standard pretrained Transformer. On the Long Range Arena (LRA) benchmark, \nfor evaluating performance on long sequences, our method achieves results consistent with softmax self-attention but with sizable \nspeed-ups and memory savings and often outperforms other efficient self-attention methods. Our code is available at this https URL*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The original code can be found [here](https://github.com/mlpen/YOSO).\n\n## Usage tips\n\n- The YOSO attention algorithm is implemented through custom CUDA kernels, functions written in CUDA C++ that can be executed multiple times\nin parallel on a GPU.\n- The kernels provide a `fast_hash` function, which approximates the random projections of the queries and keys using the Fast Hadamard Transform. Using these\nhash codes, the `lsh_cumulation` function approximates self-attention via LSH-based Bernoulli sampling.\n- To use the custom kernels, the user should set `config.use_expectation = False`. To ensure that the kernels are compiled successfully, \nthe user must install the correct version of PyTorch and cudatoolkit. By default, `config.use_expectation = True`, which uses YOSO-E and \ndoes not require compiling CUDA kernels.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/yoso_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> YOSO Attention Algorithm. Taken from the <a href=\"https://arxiv.org/abs/2111.09714\">original paper</a>.</small>\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## YosoConfig\n\n[[autodoc]] YosoConfig\n\n## YosoModel\n\n[[autodoc]] YosoModel\n    - forward\n\n## YosoForMaskedLM\n\n[[autodoc]] YosoForMaskedLM\n    - forward\n\n## YosoForSequenceClassification\n\n[[autodoc]] YosoForSequenceClassification\n    - forward\n\n## YosoForMultipleChoice\n\n[[autodoc]] YosoForMultipleChoice\n    - forward\n\n## YosoForTokenClassification\n\n[[autodoc]] YosoForTokenClassification\n    - forward\n\n## YosoForQuestionAnswering\n\n[[autodoc]] YosoForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a27aa041-9771-45d8-b8e5-1852f85f7006
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trajectory_transformer", "title": "Trajectory Transformer", "tokens": 686, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a27aa041-9771-45d8-b8e5-1852f85f7006\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trajectory_transformer\", \"title\": \"Trajectory Transformer\", \"tokens\": 686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c2ff53ad-b519-49e3-bf16-481d42fd5356\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trajectory_transformer\", \"title\": \"Trajectory Transformer\", \"tokens\": 686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"506c74b2d7f61e1d48d51094b8baa66f659140e9463a343152cfa7870bb29228\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2444, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c2ff53ad-b519-49e3-bf16-481d42fd5356", "doc_id": "c2ff53ad-b519-49e3-bf16-481d42fd5356", "ref_doc_id": "c2ff53ad-b519-49e3-bf16-481d42fd5356", "chroma:document": "# Trajectory Transformer\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, so we won't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe Trajectory Transformer model was proposed in [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://arxiv.org/abs/2106.02039)  by Michael Janner, Qiyang Li, Sergey Levine.\n\nThe abstract from the paper is the following:\n\n*Reinforcement learning (RL) is typically concerned with estimating stationary policies or single-step models,\nleveraging the Markov property to factorize problems in time. However, we can also view RL as a generic sequence\nmodeling problem, with the goal being to produce a sequence of actions that leads to a sequence of high rewards.\nViewed in this way, it is tempting to consider whether high-capacity sequence prediction models that work well\nin other domains, such as natural-language processing, can also provide effective solutions to the RL problem.\nTo this end, we explore how RL can be tackled with the tools of sequence modeling, using a Transformer architecture\nto model distributions over trajectories and repurposing beam search as a planning algorithm. Framing RL as sequence\nmodeling problem simplifies a range of design decisions, allowing us to dispense with many of the components common\nin offline RL algorithms. We demonstrate the flexibility of this approach across long-horizon dynamics prediction,\nimitation learning, goal-conditioned RL, and offline RL. Further, we show that this approach can be combined with\nexisting model-free algorithms to yield a state-of-the-art planner in sparse-reward, long-horizon tasks.*\n\nThis model was contributed by [CarlCochet](https://huggingface.co/CarlCochet). The original code can be found [here](https://github.com/jannerm/trajectory-transformer).\n\n## Usage tips\n\nThis Transformer is used for deep reinforcement learning. To use it, you need to create sequences from\nactions, states and rewards from all previous timesteps. This model will treat all these elements together\nas one big sequence (a trajectory).\n\n## TrajectoryTransformerConfig\n\n[[autodoc]] TrajectoryTransformerConfig\n\n## TrajectoryTransformerModel\n\n[[autodoc]] TrajectoryTransformerModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 766985f0-52cc-4a69-bf3b-6d7e6d73355f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/stablelm", "title": "StableLM", "tokens": 1230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"766985f0-52cc-4a69-bf3b-6d7e6d73355f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"151f4624-929d-43e2-a069-fbd67bfbf5d1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a01be8c1609bbe8d3cf8130e6956ee2fc02ea162f37a7b2905adcb350e44f118\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78920d24-7be4-4d61-b2a3-e681235a23a5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9bf9af452191f5c956f24d5a6f0a7709ac153d838c96d677c68642a3f30968e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2504, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "151f4624-929d-43e2-a069-fbd67bfbf5d1", "doc_id": "151f4624-929d-43e2-a069-fbd67bfbf5d1", "ref_doc_id": "151f4624-929d-43e2-a069-fbd67bfbf5d1", "chroma:document": "# StableLM\n\n## Overview\n\n`StableLM 3B 4E1T` was proposed in [`StableLM 3B 4E1T`: Technical Report](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo) by Stability AI and is the first model in a series of multi-epoch pre-trained language models.\n\n### Model Details\n\n`StableLM 3B 4E1T` is a decoder-only base language model pre-trained on 1 trillion tokens of diverse English and code datasets for four epochs.\nThe model architecture is transformer-based with partial Rotary Position Embeddings, SwiGLU activation, LayerNorm, etc.\n\nWe also provide `StableLM Zephyr 3B`, an instruction fine-tuned version of the model that can be used for chat-based applications.\n\n### Usage Tips\n\n- The architecture is similar to LLaMA but with RoPE applied to 25% of head embedding dimensions, LayerNorm instead of RMSNorm, and optional QKV bias terms.\n- `StableLM 3B 4E1T`-based models uses the same tokenizer as [`GPTNeoXTokenizerFast`].\n\n`StableLM 3B 4E1T` and `StableLM Zephyr 3B` can be found on the [Huggingface Hub](https://huggingface.co/stabilityai)\n\nThe following code snippet demonstrates how to use `StableLM 3B 4E1T` for inference:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> set_seed(0)\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> model_inputs = tokenizer(\"The weather is always wonderful in\", return_tensors=\"pt\").to(model.device)\n\n>>> generated_ids = model.generate(**model_inputs, max_length=32, do_sample=True)\n>>> responses = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n>>> responses\n['The weather is always wonderful in Costa Rica, which makes it a prime destination for retirees. That\u2019s where the Pensionado program comes in, offering']\n```\n\n## Combining StableLM and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention v2.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nAlso make sure that your hardware is compatible with Flash-Attention 2. Read more about it in the official documentation of the [`flash-attn`](https://github.com/Dao-AILab/flash-attention) repository. Note: you must load your model in half-precision (e.g. `torch.bfloat16`)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 78920d24-7be4-4d61-b2a3-e681235a23a5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/stablelm", "title": "StableLM", "tokens": 1230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"78920d24-7be4-4d61-b2a3-e681235a23a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"151f4624-929d-43e2-a069-fbd67bfbf5d1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a01be8c1609bbe8d3cf8130e6956ee2fc02ea162f37a7b2905adcb350e44f118\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"766985f0-52cc-4a69-bf3b-6d7e6d73355f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"69fed2cabec6f86e38d713564368a6f4b3aded74889489965202fb290142f65b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1040, \"end_char_idx\": 3907, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "151f4624-929d-43e2-a069-fbd67bfbf5d1", "doc_id": "151f4624-929d-43e2-a069-fbd67bfbf5d1", "ref_doc_id": "151f4624-929d-43e2-a069-fbd67bfbf5d1", "chroma:document": "`StableLM 3B 4E1T` and `StableLM Zephyr 3B` can be found on the [Huggingface Hub](https://huggingface.co/stabilityai)\n\nThe following code snippet demonstrates how to use `StableLM 3B 4E1T` for inference:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> set_seed(0)\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> model_inputs = tokenizer(\"The weather is always wonderful in\", return_tensors=\"pt\").to(model.device)\n\n>>> generated_ids = model.generate(**model_inputs, max_length=32, do_sample=True)\n>>> responses = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n>>> responses\n['The weather is always wonderful in Costa Rica, which makes it a prime destination for retirees. That\u2019s where the Pensionado program comes in, offering']\n```\n\n## Combining StableLM and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention v2.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nAlso make sure that your hardware is compatible with Flash-Attention 2. Read more about it in the official documentation of the [`flash-attn`](https://github.com/Dao-AILab/flash-attention) repository. Note: you must load your model in half-precision (e.g. `torch.bfloat16`).\n\nNow, to run the model with Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> set_seed(0)\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"stabilityai/stablelm-3b-4e1t\", torch_dtype=torch.bfloat16, attn_implementation=\"flash_attention_2\")  # doctest: +SKIP\n>>> model.to(device)  # doctest: +SKIP\n\n>>> model_inputs = tokenizer(\"The weather is always wonderful in\", return_tensors=\"pt\").to(model.device)\n\n>>> generated_ids = model.generate(**model_inputs, max_length=32, do_sample=True)  # doctest: +SKIP\n>>> responses = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)  # doctest: +SKIP\n>>> responses  # doctest: +SKIP\n['The weather is always wonderful in Costa Rica, which makes it a prime destination for retirees. That\u2019s where the Pensionado program comes in, offering']\n```\n\n\n## StableLmConfig\n\n[[autodoc]] StableLmConfig\n\n## StableLmModel\n\n[[autodoc]] StableLmModel\n    - forward\n\n## StableLmForCausalLM\n\n[[autodoc]] StableLmForCausalLM\n    - forward\n\n## StableLmForSequenceClassification\n\n[[autodoc]] StableLmForSequenceClassification\n    - forward\n\n## StableLmForTokenClassification\n\n[[autodoc]] StableLmForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 03a669b8-49cf-4f5b-8f09-cf6f235f5bad
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bertweet", "title": "BERTweet", "tokens": 666, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"03a669b8-49cf-4f5b-8f09-cf6f235f5bad\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bertweet\", \"title\": \"BERTweet\", \"tokens\": 666, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"56a91556-6042-4340-a614-5567d7dd8a9f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bertweet\", \"title\": \"BERTweet\", \"tokens\": 666, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fc314379a06431c93449e67d7e1f50fe1cd38654cc7b09d96cc5ef0784da5044\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2009, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "56a91556-6042-4340-a614-5567d7dd8a9f", "doc_id": "56a91556-6042-4340-a614-5567d7dd8a9f", "ref_doc_id": "56a91556-6042-4340-a614-5567d7dd8a9f", "chroma:document": "# BERTweet\n\n## Overview\n\nThe BERTweet model was proposed in [BERTweet: A pre-trained language model for English Tweets](https://www.aclweb.org/anthology/2020.emnlp-demos.2.pdf) by Dat Quoc Nguyen, Thanh Vu, Anh Tuan Nguyen.\n\nThe abstract from the paper is the following:\n\n*We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having\nthe same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et\nal., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al.,\n2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks:\nPart-of-speech tagging, Named-entity recognition and text classification.*\n\nThis model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BERTweet).\n\n## Usage example\n\n```python\n>>> import torch\n>>> from transformers import AutoModel, AutoTokenizer\n\n>>> bertweet = AutoModel.from_pretrained(\"vinai/bertweet-base\")\n\n>>> # For transformers v4.x+:\n>>> tokenizer = AutoTokenizer.from_pretrained(\"vinai/bertweet-base\", use_fast=False)\n\n>>> # For transformers v3.x:\n>>> # tokenizer = AutoTokenizer.from_pretrained(\"vinai/bertweet-base\")\n\n>>> # INPUT TWEET IS ALREADY NORMALIZED!\n>>> line = \"SC has first two presumptive cases of coronavirus , DHEC confirms HTTPURL via @USER :cry:\"\n\n>>> input_ids = torch.tensor([tokenizer.encode(line)])\n\n>>> with torch.no_grad():\n...     features = bertweet(input_ids)  # Models outputs are now tuples\n\n>>> # With TensorFlow 2.0+:\n>>> # from transformers import TFAutoModel\n>>> # bertweet = TFAutoModel.from_pretrained(\"vinai/bertweet-base\")\n```\n\n<Tip> \n\nThis implementation is the same as BERT, except for tokenization method. Refer to [BERT documentation](bert) for \nAPI reference information.  \n\n</Tip>\n\n## BertweetTokenizer\n\n[[autodoc]] BertweetTokenizer"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 43b65c13-c1ba-47fc-9f9f-5ee91cfecbfd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bridgetower", "title": "BridgeTower", "tokens": 1905, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"43b65c13-c1ba-47fc-9f9f-5ee91cfecbfd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6841d79ca12f96081c0fe3c271f21df86dfd8fd553a986006d9e9126ae64da13\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1835ebbb-e05d-489f-8eda-ee4b0ca47baf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ba1302c65f57690c858d4e6bdddd72a9d6381e9f5c06d1bc499de11740d9bf84\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3530, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "doc_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "ref_doc_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "chroma:document": "# BridgeTower\n\n## Overview\n\nThe BridgeTower model was proposed in [BridgeTower: Building Bridges Between Encoders in Vision-Language Representative Learning](https://arxiv.org/abs/2206.08657) by Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan. The goal of this model is to build a\nbridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder thus achieving remarkable performance on various downstream tasks with almost negligible additional performance and computational costs.\n\nThis paper has been accepted to the [AAAI'23](https://aaai.org/Conferences/AAAI-23/) conference. \n\nThe abstract from the paper is the following:\n\n*Vision-Language (VL) models with the TWO-TOWER architecture have dominated visual-language representation learning in recent years.\nCurrent VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder.\nBoth approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BRIDGETOWER, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the crossmodal encoder.\nThis enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BRIDGETOWER achieves state-of-the-art performance on various downstream vision-language tasks.\nIn particular, on the VQAv2 test-std set, BRIDGETOWER achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs.\nNotably, when further scaling the model, BRIDGETOWER achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/bridgetower_architecture%20.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> BridgeTower architecture. Taken from the <a href=\"https://arxiv.org/abs/2206.08657\">original paper.</a> </small>\n\nThis model was contributed by [Anahita Bhiwandiwalla](https://huggingface.co/anahita-b), [Tiep Le](https://huggingface.co/Tile) and [Shaoyen Tseng](https://huggingface.co/shaoyent). The original code can be found [here](https://github.com/microsoft/BridgeTower).\n\n## Usage tips and examples\n\nBridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers.\nThe goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder.\nIn principle, one can apply any visual, textual or cross-modal encoder in the proposed architecture.\n\nThe [`BridgeTowerProcessor`] wraps [`RobertaTokenizer`] and [`BridgeTowerImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run contrastive learning using [`BridgeTowerProcessor`] and [`BridgeTowerForContrastiveLearning`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1835ebbb-e05d-489f-8eda-ee4b0ca47baf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bridgetower", "title": "BridgeTower", "tokens": 1905, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1835ebbb-e05d-489f-8eda-ee4b0ca47baf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6841d79ca12f96081c0fe3c271f21df86dfd8fd553a986006d9e9126ae64da13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"43b65c13-c1ba-47fc-9f9f-5ee91cfecbfd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6161fbd2f75e66dd4ea5d185a381966ba17645ab63697a32e74755b60caa8c07\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"919d6e54-7174-4070-9976-b6aa53ce6353\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"638c9bcc3d6d1d32dae21daf4f2305f02b475b9777f959b10d977fb235f55080\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1792, \"end_char_idx\": 4456, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "doc_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "ref_doc_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "chroma:document": "In particular, on the VQAv2 test-std set, BRIDGETOWER achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs.\nNotably, when further scaling the model, BRIDGETOWER achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/bridgetower_architecture%20.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> BridgeTower architecture. Taken from the <a href=\"https://arxiv.org/abs/2206.08657\">original paper.</a> </small>\n\nThis model was contributed by [Anahita Bhiwandiwalla](https://huggingface.co/anahita-b), [Tiep Le](https://huggingface.co/Tile) and [Shaoyen Tseng](https://huggingface.co/shaoyent). The original code can be found [here](https://github.com/microsoft/BridgeTower).\n\n## Usage tips and examples\n\nBridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers.\nThe goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder.\nIn principle, one can apply any visual, textual or cross-modal encoder in the proposed architecture.\n\nThe [`BridgeTowerProcessor`] wraps [`RobertaTokenizer`] and [`BridgeTowerImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run contrastive learning using [`BridgeTowerProcessor`] and [`BridgeTowerForContrastiveLearning`].\n```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning\n>>> import requests\n>>> from PIL import Image\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [\"An image of two cats chilling on a couch\", \"A football player scoring a goal\"]\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-large-itm-mlm-itc\")\n>>> model = BridgeTowerForContrastiveLearning.from_pretrained(\"BridgeTower/bridgetower-large-itm-mlm-itc\")\n\n>>> # forward pass\n>>> scores = dict()\n>>> for text in texts:\n...     # prepare inputs\n...     encoding = processor(image, text, return_tensors=\"pt\")\n...     outputs = model(**encoding)\n...     scores[text] = outputs\n```\n\nThe following example shows how to run image-text retrieval using [`BridgeTowerProcessor`] and [`BridgeTowerForImageAndTextRetrieval`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 919d6e54-7174-4070-9976-b6aa53ce6353
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bridgetower", "title": "BridgeTower", "tokens": 1905, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"919d6e54-7174-4070-9976-b6aa53ce6353\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6841d79ca12f96081c0fe3c271f21df86dfd8fd553a986006d9e9126ae64da13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1835ebbb-e05d-489f-8eda-ee4b0ca47baf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8877df242c18e0e61f084467e08b391e4685008f650dcdccaaa21e263803bf56\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"455748d3-2276-43ef-bb81-411ef8a77ae5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0144efbf4565b3a177d05da87af3406c60e3901d8d46b50eaed090d1bb3e7050\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2687, \"end_char_idx\": 5387, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "doc_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "ref_doc_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "chroma:document": "The original code can be found [here](https://github.com/microsoft/BridgeTower).\n\n## Usage tips and examples\n\nBridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers.\nThe goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder.\nIn principle, one can apply any visual, textual or cross-modal encoder in the proposed architecture.\n\nThe [`BridgeTowerProcessor`] wraps [`RobertaTokenizer`] and [`BridgeTowerImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run contrastive learning using [`BridgeTowerProcessor`] and [`BridgeTowerForContrastiveLearning`].\n```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning\n>>> import requests\n>>> from PIL import Image\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [\"An image of two cats chilling on a couch\", \"A football player scoring a goal\"]\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-large-itm-mlm-itc\")\n>>> model = BridgeTowerForContrastiveLearning.from_pretrained(\"BridgeTower/bridgetower-large-itm-mlm-itc\")\n\n>>> # forward pass\n>>> scores = dict()\n>>> for text in texts:\n...     # prepare inputs\n...     encoding = processor(image, text, return_tensors=\"pt\")\n...     outputs = model(**encoding)\n...     scores[text] = outputs\n```\n\nThe following example shows how to run image-text retrieval using [`BridgeTowerProcessor`] and [`BridgeTowerForImageAndTextRetrieval`].\n```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval\n>>> import requests\n>>> from PIL import Image\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [\"An image of two cats chilling on a couch\", \"A football player scoring a goal\"]\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n>>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n\n>>> # forward pass\n>>> scores = dict()\n>>> for text in texts:\n...     # prepare inputs\n...     encoding = processor(image, text, return_tensors=\"pt\")\n...     outputs = model(**encoding)\n...     scores[text] = outputs.logits[0, 1].item()\n```\n\nThe following example shows how to run masked language modeling using [`BridgeTowerProcessor`] and [`BridgeTowerForMaskedLM`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 455748d3-2276-43ef-bb81-411ef8a77ae5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bridgetower", "title": "BridgeTower", "tokens": 1905, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"455748d3-2276-43ef-bb81-411ef8a77ae5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6841d79ca12f96081c0fe3c271f21df86dfd8fd553a986006d9e9126ae64da13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"919d6e54-7174-4070-9976-b6aa53ce6353\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bff0c92981345ff5bde68e52f33e381eba10f1577a9cb27a89693427e0a688d7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4457, \"end_char_idx\": 7476, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "doc_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "ref_doc_id": "d97a58a5-14ca-4cc4-8dc8-e9b45acb37a3", "chroma:document": "```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval\n>>> import requests\n>>> from PIL import Image\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [\"An image of two cats chilling on a couch\", \"A football player scoring a goal\"]\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n>>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n\n>>> # forward pass\n>>> scores = dict()\n>>> for text in texts:\n...     # prepare inputs\n...     encoding = processor(image, text, return_tensors=\"pt\")\n...     outputs = model(**encoding)\n...     scores[text] = outputs.logits[0, 1].item()\n```\n\nThe following example shows how to run masked language modeling using [`BridgeTowerProcessor`] and [`BridgeTowerForMaskedLM`].\n\n```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM\n>>> from PIL import Image\n>>> import requests\n\n>>> url = \"http://images.cocodataset.org/val2017/000000360943.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw).convert(\"RGB\")\n>>> text = \"a <mask> looking out of the window\"\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n>>> model = BridgeTowerForMaskedLM.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n\n>>> # prepare inputs\n>>> encoding = processor(image, text, return_tensors=\"pt\")\n\n>>> # forward pass\n>>> outputs = model(**encoding)\n\n>>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist())\n\n>>> print(results)\n.a cat looking out of the window.\n```\n\nTips:\n\n- This implementation of BridgeTower uses [`RobertaTokenizer`] to generate text embeddings and OpenAI's CLIP/ViT model to compute visual embeddings.\n- Checkpoints for pre-trained [bridgeTower-base](https://huggingface.co/BridgeTower/bridgetower-base) and [bridgetower masked language modeling and image text matching](https://huggingface.co/BridgeTower/bridgetower-base-itm-mlm) are released.\n- Please refer to [Table 5](https://arxiv.org/pdf/2206.08657.pdf) for BridgeTower's performance on Image Retrieval and other down stream tasks.\n- The PyTorch version of this model is only available in torch 1.10 and higher.\n\n\n## BridgeTowerConfig\n\n[[autodoc]] BridgeTowerConfig\n\n## BridgeTowerTextConfig\n\n[[autodoc]] BridgeTowerTextConfig\n\n## BridgeTowerVisionConfig\n\n[[autodoc]] BridgeTowerVisionConfig\n\n## BridgeTowerImageProcessor\n\n[[autodoc]] BridgeTowerImageProcessor\n    - preprocess\n\n## BridgeTowerProcessor\n\n[[autodoc]] BridgeTowerProcessor\n    - __call__\n\n## BridgeTowerModel\n\n[[autodoc]] BridgeTowerModel\n    - forward\n\n## BridgeTowerForContrastiveLearning\n\n[[autodoc]] BridgeTowerForContrastiveLearning\n    - forward\n\n## BridgeTowerForMaskedLM\n\n[[autodoc]] BridgeTowerForMaskedLM\n    - forward\n\n## BridgeTowerForImageAndTextRetrieval\n\n[[autodoc]] BridgeTowerForImageAndTextRetrieval\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a79e1a8f-1ecc-41d5-9cf9-8565bad7156d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a79e1a8f-1ecc-41d5-9cf9-8565bad7156d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9dbcc63e-0662-4369-a761-66f07dfaf599\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d6a12c34c4d42167f38c89ca26c2813031a44cfe6bab326404fe76c86ad60c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0eaa804f-1844-465b-97aa-9116735110b5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bc63cbd81fe2cf1a2a224adca8ec157427f4ec4c02ae8d8773874ef45f55c414\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3053, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "ref_doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "chroma:document": "# BART\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=bart\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-bart-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/bart-large-mnli\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Bart model was proposed in [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation,\nTranslation, and Comprehension](https://arxiv.org/abs/1910.13461) by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan\nGhazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer on 29 Oct, 2019.\n\nAccording to the abstract,\n\n- Bart uses a standard seq2seq/machine translation architecture with a bidirectional encoder (like BERT) and a\n  left-to-right decoder (like GPT).\n- The pretraining task involves randomly shuffling the order of the original sentences and a novel in-filling scheme,\n  where spans of text are replaced with a single mask token.\n- BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It\n  matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new\n  state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains\n  of up to 6 ROUGE.\n\nThis model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/bart).\n\n## Usage tips:\n\n- BART is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- Sequence-to-sequence model with an encoder and a decoder. Encoder is fed a corrupted version of the tokens, decoder is fed the original tokens (but has a mask to hide the future words like a regular transformers decoder). A composition of the following transformations are applied on the pretraining tasks for the encoder:\n\n  * mask random tokens (like in BERT)\n  * delete random tokens\n  * mask a span of k tokens with a single mask token (a span of 0 tokens is an insertion of a mask token)\n  * permute sentences\n  * rotate the document to make it start at a specific token\n\n## Implementation Notes\n\n- Bart doesn't use `token_type_ids` for sequence classification. Use [`BartTokenizer`] or\n  [`~BartTokenizer.encode`] to get the proper splitting.\n- The forward pass of [`BartModel`] will create the `decoder_input_ids` if they are not passed.\n  This is different than some other modeling APIs. A typical use case of this feature is mask filling.\n- Model predictions are intended to be identical to the original implementation when\n  `forced_bos_token_id=0`. This only works, however, if the string you pass to\n  [`fairseq.encode`] starts with a space.\n- [`~generation.GenerationMixin.generate`] should be used for conditional generation tasks like\n  summarization, see the example in that docstrings."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0eaa804f-1844-465b-97aa-9116735110b5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0eaa804f-1844-465b-97aa-9116735110b5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9dbcc63e-0662-4369-a761-66f07dfaf599\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d6a12c34c4d42167f38c89ca26c2813031a44cfe6bab326404fe76c86ad60c3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a79e1a8f-1ecc-41d5-9cf9-8565bad7156d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d68e6ec6a66741b3d3b810d6a6f6df8530cdbbc7c2e4d1cf59729799c385e2ed\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"36231f8e-b73e-4f6c-b687-96d5b128d903\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"08fdad219731f1b678ae7f752074bc1dc2135181b463433064b3ebb8cfb34f9d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1423, \"end_char_idx\": 4668, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "ref_doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "chroma:document": "This model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/bart).\n\n## Usage tips:\n\n- BART is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- Sequence-to-sequence model with an encoder and a decoder. Encoder is fed a corrupted version of the tokens, decoder is fed the original tokens (but has a mask to hide the future words like a regular transformers decoder). A composition of the following transformations are applied on the pretraining tasks for the encoder:\n\n  * mask random tokens (like in BERT)\n  * delete random tokens\n  * mask a span of k tokens with a single mask token (a span of 0 tokens is an insertion of a mask token)\n  * permute sentences\n  * rotate the document to make it start at a specific token\n\n## Implementation Notes\n\n- Bart doesn't use `token_type_ids` for sequence classification. Use [`BartTokenizer`] or\n  [`~BartTokenizer.encode`] to get the proper splitting.\n- The forward pass of [`BartModel`] will create the `decoder_input_ids` if they are not passed.\n  This is different than some other modeling APIs. A typical use case of this feature is mask filling.\n- Model predictions are intended to be identical to the original implementation when\n  `forced_bos_token_id=0`. This only works, however, if the string you pass to\n  [`fairseq.encode`] starts with a space.\n- [`~generation.GenerationMixin.generate`] should be used for conditional generation tasks like\n  summarization, see the example in that docstrings.\n- Models that load the *facebook/bart-large-cnn* weights will not have a `mask_token_id`, or be able to perform\n  mask-filling tasks.\n\n## Mask Filling\n\nThe `facebook/bart-base` and `facebook/bart-large` checkpoints can be used to fill multi-token masks.\n\n```python\nfrom transformers import BartForConditionalGeneration, BartTokenizer\n\nmodel = BartForConditionalGeneration.from_pretrained(\"facebook/bart-large\", forced_bos_token_id=0)\ntok = BartTokenizer.from_pretrained(\"facebook/bart-large\")\nexample_english_phrase = \"UN Chief Says There Is No <mask> in Syria\"\nbatch = tok(example_english_phrase, return_tensors=\"pt\")\ngenerated_ids = model.generate(batch[\"input_ids\"])\nassert tok.batch_decode(generated_ids, skip_special_tokens=True) == [\n    \"UN Chief Says There Is No Plan to Stop Chemical Weapons in Syria\"\n]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BART. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"summarization\"/>\n\n- A blog post on [Distributed Training: Train BART/T5 for Summarization using \ud83e\udd17 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).\n- A notebook on how to [finetune BART for summarization with fastai using blurr](https://colab.research.google.com/github/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 36231f8e-b73e-4f6c-b687-96d5b128d903
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"36231f8e-b73e-4f6c-b687-96d5b128d903\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9dbcc63e-0662-4369-a761-66f07dfaf599\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d6a12c34c4d42167f38c89ca26c2813031a44cfe6bab326404fe76c86ad60c3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0eaa804f-1844-465b-97aa-9116735110b5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6398508143fd500c5b884614a5c79d31a52cc9c2678c9fb5fe0a1193996651e2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"baef26ba-7644-43ad-a2ed-a7e938183c51\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4e3fe4e5cc1790648957979804f2ae6d2193fc2611b17f2b4ce141d9a077e52d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3054, \"end_char_idx\": 5969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "ref_doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "chroma:document": "- Models that load the *facebook/bart-large-cnn* weights will not have a `mask_token_id`, or be able to perform\n  mask-filling tasks.\n\n## Mask Filling\n\nThe `facebook/bart-base` and `facebook/bart-large` checkpoints can be used to fill multi-token masks.\n\n```python\nfrom transformers import BartForConditionalGeneration, BartTokenizer\n\nmodel = BartForConditionalGeneration.from_pretrained(\"facebook/bart-large\", forced_bos_token_id=0)\ntok = BartTokenizer.from_pretrained(\"facebook/bart-large\")\nexample_english_phrase = \"UN Chief Says There Is No <mask> in Syria\"\nbatch = tok(example_english_phrase, return_tensors=\"pt\")\ngenerated_ids = model.generate(batch[\"input_ids\"])\nassert tok.batch_decode(generated_ids, skip_special_tokens=True) == [\n    \"UN Chief Says There Is No Plan to Stop Chemical Weapons in Syria\"\n]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BART. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"summarization\"/>\n\n- A blog post on [Distributed Training: Train BART/T5 for Summarization using \ud83e\udd17 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).\n- A notebook on how to [finetune BART for summarization with fastai using blurr](https://colab.research.google.com/github/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune BART for summarization in two languages with Trainer class](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n- [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).\n- An example of how to train [`BartForConditionalGeneration`] with a Hugging Face `datasets` object can be found in this [forum discussion](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904)\n- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the \ud83e\udd17 Hugging Face course."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: baef26ba-7644-43ad-a2ed-a7e938183c51
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"baef26ba-7644-43ad-a2ed-a7e938183c51\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9dbcc63e-0662-4369-a761-66f07dfaf599\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d6a12c34c4d42167f38c89ca26c2813031a44cfe6bab326404fe76c86ad60c3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"36231f8e-b73e-4f6c-b687-96d5b128d903\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2ebfcb1c3b201cb9db547527043352dda2c94239eb03dea4fe920d669d25c9fb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b60f7816-e0e6-4089-972b-2bac9205a2f2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"72d14d2adb844c7a006c097fa518901f4a136517c440a5e71713d5e0774293c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4669, \"end_char_idx\": 7741, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "ref_doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "chroma:document": "\ud83c\udf0e\n- A notebook on how to [finetune BART for summarization in two languages with Trainer class](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n- [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).\n- An example of how to train [`BartForConditionalGeneration`] with a Hugging Face `datasets` object can be found in this [forum discussion](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904)\n- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the \ud83e\udd17 Hugging Face course.\n- [Summarization task guide](../tasks/summarization)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"translation\"/>\n\n- A notebook on how to [finetune mBART using Seq2SeqTrainer for Hindi to English translation](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b60f7816-e0e6-4089-972b-2bac9205a2f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b60f7816-e0e6-4089-972b-2bac9205a2f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9dbcc63e-0662-4369-a761-66f07dfaf599\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d6a12c34c4d42167f38c89ca26c2813031a44cfe6bab326404fe76c86ad60c3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"baef26ba-7644-43ad-a2ed-a7e938183c51\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53a10a9c76f793e15b8644ca8e723a1e69f38366a52e1c071f091f4c4a10bd03\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e9998450-23cb-4f17-bbc1-605761cc1d3e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6ce315a056bad78fd4f1cfc6ddc01aa745fa2f7738a06635767d60bbb3b534ff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6398, \"end_char_idx\": 8590, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "ref_doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "chroma:document": "- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"translation\"/>\n\n- A notebook on how to [finetune mBART using Seq2SeqTrainer for Hindi to English translation](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n- [Translation task guide](../tasks/translation)\n\nSee also:\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Distilled checkpoints](https://huggingface.co/models?search=distilbart) are described in this [paper](https://arxiv.org/abs/2010.13002).\n\n## BartConfig\n\n[[autodoc]] BartConfig\n    - all\n\n## BartTokenizer\n\n[[autodoc]] BartTokenizer\n    - all\n\n## BartTokenizerFast\n\n[[autodoc]] BartTokenizerFast\n    - all"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e9998450-23cb-4f17-bbc1-605761cc1d3e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e9998450-23cb-4f17-bbc1-605761cc1d3e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9dbcc63e-0662-4369-a761-66f07dfaf599\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d6a12c34c4d42167f38c89ca26c2813031a44cfe6bab326404fe76c86ad60c3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b60f7816-e0e6-4089-972b-2bac9205a2f2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e695fb8d5d20f3e7d08a856c2a9590075c7cba1b842bd5bcb3ca6a93819b1a54\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7026, \"end_char_idx\": 9801, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "ref_doc_id": "9dbcc63e-0662-4369-a761-66f07dfaf599", "chroma:document": "- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"translation\"/>\n\n- A notebook on how to [finetune mBART using Seq2SeqTrainer for Hindi to English translation](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n- [Translation task guide](../tasks/translation)\n\nSee also:\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Distilled checkpoints](https://huggingface.co/models?search=distilbart) are described in this [paper](https://arxiv.org/abs/2010.13002).\n\n## BartConfig\n\n[[autodoc]] BartConfig\n    - all\n\n## BartTokenizer\n\n[[autodoc]] BartTokenizer\n    - all\n\n## BartTokenizerFast\n\n[[autodoc]] BartTokenizerFast\n    - all\n\n\n<frameworkcontent>\n<pt>\n\n## BartModel\n\n[[autodoc]] BartModel\n    - forward\n\n## BartForConditionalGeneration\n\n[[autodoc]] BartForConditionalGeneration\n    - forward\n\n## BartForSequenceClassification\n\n[[autodoc]] BartForSequenceClassification\n    - forward\n\n## BartForQuestionAnswering\n\n[[autodoc]] BartForQuestionAnswering\n    - forward\n\n## BartForCausalLM\n\n[[autodoc]] BartForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFBartModel\n\n[[autodoc]] TFBartModel\n    - call\n\n## TFBartForConditionalGeneration\n\n[[autodoc]] TFBartForConditionalGeneration\n    - call\n\n## TFBartForSequenceClassification\n\n[[autodoc]] TFBartForSequenceClassification\n    - call\n\n</tf>\n<jax>\n\n## FlaxBartModel\n\n[[autodoc]] FlaxBartModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxBartForConditionalGeneration\n\n[[autodoc]] FlaxBartForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n## FlaxBartForSequenceClassification\n\n[[autodoc]] FlaxBartForSequenceClassification\n    - __call__\n    - encode\n    - decode\n\n## FlaxBartForQuestionAnswering\n\n[[autodoc]] FlaxBartForQuestionAnswering\n    - __call__\n    - encode\n    - decode\n\n## FlaxBartForCausalLM\n\n[[autodoc]] FlaxBartForCausalLM\n    - __call__\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fbf65aad-885b-441b-af3e-859d66b21007
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dac", "title": "DAC", "tokens": 1022, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fbf65aad-885b-441b-af3e-859d66b21007\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dac\", \"title\": \"DAC\", \"tokens\": 1022, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"afef7677-02fe-4bb4-83b7-15a770732cd5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dac\", \"title\": \"DAC\", \"tokens\": 1022, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fa2abd919fceebb0fc04344b590d0043260bd89bddefc60b8df8c95f7b6bf179\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"79d6dd95-7f79-4500-b62c-7aa04e209e4f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0c4ab82cf8a4df807354effc985c24dea206f68ae551a75dec3dae77150a3839\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2111, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "afef7677-02fe-4bb4-83b7-15a770732cd5", "doc_id": "afef7677-02fe-4bb4-83b7-15a770732cd5", "ref_doc_id": "afef7677-02fe-4bb4-83b7-15a770732cd5", "chroma:document": "# DAC\n\n## Overview\n\n\nThe DAC model was proposed in [Descript Audio Codec: High-Fidelity Audio Compression with Improved RVQGAN](https://arxiv.org/abs/2306.06546) by Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, Kundan Kumar.\n\nThe Descript Audio Codec (DAC) model is a powerful tool for compressing audio data, making it highly efficient for storage and transmission. By compressing 44.1 KHz audio into tokens at just 8kbps bandwidth, the DAC model enables high-quality audio processing while significantly reducing the data footprint. This is particularly useful in scenarios where bandwidth is limited or storage space is at a premium, such as in streaming applications, remote conferencing, and archiving large audio datasets.\n\nThe abstract from the paper is the following:\n\n*Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can compress high-dimensional natural signals into lower dimensional discrete tokens. To that end, we introduce a high-fidelity universal neural audio compression algorithm that achieves ~90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve this by combining advances in high-fidelity audio generation with better vector quantization techniques from the image domain, along with improved adversarial and reconstruction losses. We compress all domains (speech, environment, music, etc.) with a single universal model, making it widely applicable to generative modeling of all audio. We compare with competing audio compression algorithms, and find our method outperforms them significantly. We provide thorough ablations for every design choice, as well as open-source code and trained model weights. We hope our work can lay the foundation for the next generation of high-fidelity audio modeling.*\n\nThis model was contributed by [Kamil Akesbi](https://huggingface.co/kamilakesbi).\nThe original code can be found [here](https://github.com/descriptinc/descript-audio-codec/tree/main?tab=readme-ov-file)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 79d6dd95-7f79-4500-b62c-7aa04e209e4f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dac", "title": "DAC", "tokens": 1022, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"79d6dd95-7f79-4500-b62c-7aa04e209e4f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dac\", \"title\": \"DAC\", \"tokens\": 1022, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"afef7677-02fe-4bb4-83b7-15a770732cd5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dac\", \"title\": \"DAC\", \"tokens\": 1022, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fa2abd919fceebb0fc04344b590d0043260bd89bddefc60b8df8c95f7b6bf179\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fbf65aad-885b-441b-af3e-859d66b21007\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dac\", \"title\": \"DAC\", \"tokens\": 1022, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"23d1f7a61bfbf41d491649369cf02a0b92e3195fccb4ac85a10122e347e2c0c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2114, \"end_char_idx\": 3974, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "afef7677-02fe-4bb4-83b7-15a770732cd5", "doc_id": "afef7677-02fe-4bb4-83b7-15a770732cd5", "ref_doc_id": "afef7677-02fe-4bb4-83b7-15a770732cd5", "chroma:document": "## Model structure\n\nThe Descript Audio Codec (DAC) model is structured into three distinct stages:\n\n1. Encoder Model: This stage compresses the input audio, reducing its size while retaining essential information.\n2. Residual Vector Quantizer (RVQ) Model: Working in tandem with the encoder, this model quantizes the latent codes of the audio, refining the compression and ensuring high-quality reconstruction.\n3. Decoder Model: This final stage reconstructs the audio from its compressed form, restoring it to a state that closely resembles the original input.\n\n## Usage example \n\nHere is a quick example of how to encode and decode an audio using this model: \n\n```python \n>>> from datasets import load_dataset, Audio\n>>> from transformers import DacModel, AutoProcessor\n>>> librispeech_dummy = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n\n>>> model = DacModel.from_pretrained(\"descript/dac_16khz\")\n>>> processor = AutoProcessor.from_pretrained(\"descript/dac_16khz\")\n>>> librispeech_dummy = librispeech_dummy.cast_column(\"audio\", Audio(sampling_rate=processor.sampling_rate))\n>>> audio_sample = librispeech_dummy[-1][\"audio\"][\"array\"]\n>>> inputs = processor(raw_audio=audio_sample, sampling_rate=processor.sampling_rate, return_tensors=\"pt\")\n\n>>> encoder_outputs = model.encode(inputs[\"input_values\"])\n>>> # Get the intermediate audio codes\n>>> audio_codes = encoder_outputs.audio_codes\n>>> # Reconstruct the audio from its quantized representation\n>>> audio_values = model.decode(encoder_outputs.quantized_representation)\n>>> # or the equivalent with a forward pass\n>>> audio_values = model(inputs[\"input_values\"]).audio_values\n```\n\n## DacConfig\n\n[[autodoc]] DacConfig\n\n## DacFeatureExtractor\n\n[[autodoc]] DacFeatureExtractor\n    - __call__\n\n## DacModel\n\n[[autodoc]] DacModel\n    - decode\n    - encode\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0ea89992-55c9-4cb8-a247-89d0104dc8b4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapex", "title": "TAPEX", "tokens": 1845, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0ea89992-55c9-4cb8-a247-89d0104dc8b4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87ed64d5-7c96-4532-8692-0b8ae525ed33\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"89187f4979b4b5c48d93d82c138bbaf27bc22127d5885139bc7536f6f82c168c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7ed1b8b5-42c7-45ac-88c1-a7d85ba29a95\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8dc73e350e1b4acdc6765af037a8c02ccfce36faea71466bf15c3b36095737e9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3054, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "doc_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "ref_doc_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "chroma:document": "# TAPEX\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe TAPEX model was proposed in [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://arxiv.org/abs/2107.07653) by Qian Liu,\nBei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, Jian-Guang Lou. TAPEX pre-trains a BART model to solve synthetic SQL queries, after\nwhich it can be fine-tuned to answer natural language questions related to tabular data, as well as performing table fact checking. \n\nTAPEX has been fine-tuned on several datasets: \n- [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) (Sequential Question Answering by Microsoft)\n- [WTQ](https://github.com/ppasupat/WikiTableQuestions) (Wiki Table Questions by Stanford University)\n- [WikiSQL](https://github.com/salesforce/WikiSQL) (by Salesforce)\n- [TabFact](https://tabfact.github.io/) (by USCB NLP Lab).\n\nThe abstract from the paper is the following:\n\n*Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is\nstill a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we\npropose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically\nsynthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL\nexecutor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that\nTAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes improvements\non the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy\nto 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs\nand to achieve new state-of-the-art results on various downstream tasks.*\n\n## Usage tips\n\n- TAPEX is a generative (seq2seq) model. One can directly plug in the weights of TAPEX into a BART model. \n- TAPEX has checkpoints on the hub that are either pre-trained only, or fine-tuned on WTQ, SQA, WikiSQL and TabFact.\n- Sentences + tables are presented to the model as `sentence + \" \" + linearized table`. The linearized table has the following format: \n  `col: col1 | col2 | col 3 row 1 : val1 | val2 | val3 row 2 : ...`.\n- TAPEX has its own tokenizer, that allows to prepare all data for the model easily."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7ed1b8b5-42c7-45ac-88c1-a7d85ba29a95
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapex", "title": "TAPEX", "tokens": 1845, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7ed1b8b5-42c7-45ac-88c1-a7d85ba29a95\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87ed64d5-7c96-4532-8692-0b8ae525ed33\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"89187f4979b4b5c48d93d82c138bbaf27bc22127d5885139bc7536f6f82c168c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0ea89992-55c9-4cb8-a247-89d0104dc8b4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1457c4b89913f9d2049d51b34ed9ca5ea78977de8a9c0e9532ddd0571888cfbe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f152c570-0557-4387-9b78-c0a00024dd9d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b7bd654a8835e97e6660b7e6d82dcc0cd45a03fef0ced43ae8ce561bad31e008\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1480, \"end_char_idx\": 4693, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "doc_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "ref_doc_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "chroma:document": "In this paper, we\npropose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically\nsynthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL\nexecutor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that\nTAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes improvements\non the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy\nto 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs\nand to achieve new state-of-the-art results on various downstream tasks.*\n\n## Usage tips\n\n- TAPEX is a generative (seq2seq) model. One can directly plug in the weights of TAPEX into a BART model. \n- TAPEX has checkpoints on the hub that are either pre-trained only, or fine-tuned on WTQ, SQA, WikiSQL and TabFact.\n- Sentences + tables are presented to the model as `sentence + \" \" + linearized table`. The linearized table has the following format: \n  `col: col1 | col2 | col 3 row 1 : val1 | val2 | val3 row 2 : ...`.\n- TAPEX has its own tokenizer, that allows to prepare all data for the model easily. One can pass Pandas DataFrames and strings to the tokenizer,\n  and it will automatically create the `input_ids` and `attention_mask` (as shown in the usage examples below). \n\n### Usage: inference\n\nBelow, we illustrate how to use TAPEX for table question answering. As one can see, one can directly plug in the weights of TAPEX into a BART model.\nWe use the [Auto API](auto), which will automatically instantiate the appropriate tokenizer ([`TapexTokenizer`]) and model ([`BartForConditionalGeneration`]) for us,\nbased on the configuration file of the checkpoint on the hub.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import pandas as pd\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/tapex-large-finetuned-wtq\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"microsoft/tapex-large-finetuned-wtq\")\n\n>>> # prepare table + question\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> question = \"how many movies does Leonardo Di Caprio have?\"\n\n>>> encoding = tokenizer(table, question, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> predicted_answer = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> print(predicted_answer)\n53\n```\n\nNote that [`TapexTokenizer`] also supports batched inference. Hence, one can provide a batch of different tables/questions, or a batch of a single table\nand multiple questions, or a batch of a single query and multiple tables."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f152c570-0557-4387-9b78-c0a00024dd9d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapex", "title": "TAPEX", "tokens": 1845, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f152c570-0557-4387-9b78-c0a00024dd9d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87ed64d5-7c96-4532-8692-0b8ae525ed33\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"89187f4979b4b5c48d93d82c138bbaf27bc22127d5885139bc7536f6f82c168c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7ed1b8b5-42c7-45ac-88c1-a7d85ba29a95\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d003ebe692fd21ea4445fba7718c419d4be20e1e004f558619a32d5ee9916008\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"195f809f-54d7-4806-a1b7-afd9aa8b7919\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"adcbf080a422dcce3158cd1d53c42609d741a86706bf00096f88329952cc07d1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3055, \"end_char_idx\": 5817, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "doc_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "ref_doc_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "chroma:document": "One can pass Pandas DataFrames and strings to the tokenizer,\n  and it will automatically create the `input_ids` and `attention_mask` (as shown in the usage examples below). \n\n### Usage: inference\n\nBelow, we illustrate how to use TAPEX for table question answering. As one can see, one can directly plug in the weights of TAPEX into a BART model.\nWe use the [Auto API](auto), which will automatically instantiate the appropriate tokenizer ([`TapexTokenizer`]) and model ([`BartForConditionalGeneration`]) for us,\nbased on the configuration file of the checkpoint on the hub.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import pandas as pd\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/tapex-large-finetuned-wtq\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"microsoft/tapex-large-finetuned-wtq\")\n\n>>> # prepare table + question\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> question = \"how many movies does Leonardo Di Caprio have?\"\n\n>>> encoding = tokenizer(table, question, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> predicted_answer = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> print(predicted_answer)\n53\n```\n\nNote that [`TapexTokenizer`] also supports batched inference. Hence, one can provide a batch of different tables/questions, or a batch of a single table\nand multiple questions, or a batch of a single query and multiple tables. Let's illustrate this:\n\n```python\n>>> # prepare table + question\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> questions = [\n...     \"how many movies does Leonardo Di Caprio have?\",\n...     \"which actor has 69 movies?\",\n...     \"what's the first name of the actor who has 87 movies?\",\n... ]\n>>> encoding = tokenizer(table, questions, padding=True, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n[' 53', ' george clooney', ' brad pitt']\n```\n\nIn case one wants to do table verification (i.e. the task of determining whether a given sentence is supported or refuted by the contents\nof a table), one can instantiate a [`BartForSequenceClassification`] model. TAPEX has checkpoints on the hub fine-tuned on TabFact, an important\nbenchmark for table fact checking (it achieves 84% accuracy). The code example below again leverages the [Auto API](auto)."}


