# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8f58e40e-3725-47e5-b0b9-ab2e85c984df
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yolos", "title": "YOLOS", "tokens": 1419, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8f58e40e-3725-47e5-b0b9-ab2e85c984df\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c31ce59b-d53f-45c4-9a66-a81a2d10817f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15e603461d5b610e0da53196578808c3a31bc7cd2f3d46b2d21862f97944ca20\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"88ec6d83-42d8-45f6-8578-8f6945916858\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6ccd7b73081886a71771ece4cd30c8dd5c43fed6766da0e9b06fc43122d0fafa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8b25ab92-fdb2-450d-84f0-b2ac0a780162\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b06da1c0fb6ee413ad116d401252b45a2600973d2696bb43f10033d73bc1a34a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1614, \"end_char_idx\": 4956, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c31ce59b-d53f-45c4-9a66-a81a2d10817f", "doc_id": "c31ce59b-d53f-45c4-9a66-a81a2d10817f", "ref_doc_id": "c31ce59b-d53f-45c4-9a66-a81a2d10817f", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2106.00666\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/hustvl/YOLOS).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import AutoModelForObjectDetection\nmodel = AutoModelForObjectDetection.from_pretrained(\"hustvl/yolos-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `hustvl/yolos-base` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                       106 |                                        76 |                      1.39 |\n|            2 |                                       154 |                                        90 |                      1.71 |\n|            4 |                                       222 |                                       116 |                      1.91 |\n|            8 |                                       368 |                                       168 |                      2.19 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with YOLOS.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- All example notebooks illustrating inference + fine-tuning [`YolosForObjectDetection`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/YOLOS).\n- Scripts for finetuning [`YolosForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<Tip>\n\nUse [`YolosImageProcessor`] for preparing images (and optional targets) for the model. Contrary to [DETR](detr), YOLOS doesn't require a `pixel_mask` to be created."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8b25ab92-fdb2-450d-84f0-b2ac0a780162
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yolos", "title": "YOLOS", "tokens": 1419, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8b25ab92-fdb2-450d-84f0-b2ac0a780162\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c31ce59b-d53f-45c4-9a66-a81a2d10817f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15e603461d5b610e0da53196578808c3a31bc7cd2f3d46b2d21862f97944ca20\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8f58e40e-3725-47e5-b0b9-ab2e85c984df\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8cf7191a8d9c82f10f4bab69f76f567b509eaee7e77b88a43c87a7c58b5f1d97\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3070, \"end_char_idx\": 5379, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c31ce59b-d53f-45c4-9a66-a81a2d10817f", "doc_id": "c31ce59b-d53f-45c4-9a66-a81a2d10817f", "ref_doc_id": "c31ce59b-d53f-45c4-9a66-a81a2d10817f", "chroma:document": "|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                       106 |                                        76 |                      1.39 |\n|            2 |                                       154 |                                        90 |                      1.71 |\n|            4 |                                       222 |                                       116 |                      1.91 |\n|            8 |                                       368 |                                       168 |                      2.19 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with YOLOS.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- All example notebooks illustrating inference + fine-tuning [`YolosForObjectDetection`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/YOLOS).\n- Scripts for finetuning [`YolosForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<Tip>\n\nUse [`YolosImageProcessor`] for preparing images (and optional targets) for the model. Contrary to [DETR](detr), YOLOS doesn't require a `pixel_mask` to be created.\n\n</Tip>\n\n## YolosConfig\n\n[[autodoc]] YolosConfig\n\n## YolosImageProcessor\n\n[[autodoc]] YolosImageProcessor\n    - preprocess\n    - pad\n    - post_process_object_detection\n\n## YolosFeatureExtractor\n\n[[autodoc]] YolosFeatureExtractor\n    - __call__\n    - pad\n    - post_process_object_detection\n\n## YolosModel\n\n[[autodoc]] YolosModel\n    - forward\n\n## YolosForObjectDetection\n\n[[autodoc]] YolosForObjectDetection\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6f00947c-bb43-415e-a9cb-8da1ed3424a3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6f00947c-bb43-415e-a9cb-8da1ed3424a3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97fee460-6c8e-4982-bc78-2b437b62f5ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1a78d92a-948b-4beb-9941-1cb9c60dadc7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bc343e37f3b930bfc8bd338c012550f5bcfac624bd82d5dc3bfbbc05d795dbd9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1898, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "ref_doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "chroma:document": "# Mistral\n\n## Overview\n\nMistral was introduced in the [this blogpost](https://mistral.ai/news/announcing-mistral-7b/) by Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, L\u00e9lio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth\u00e9e Lacroix, William El Sayed.\n\nThe introduction of the blog post says:\n\n*Mistral AI team is proud to release Mistral 7B, the most powerful language model for its size to date.*\n\nMistral-7B is the first large language model (LLM) released by [mistral.ai](https://mistral.ai/).\n\n### Architectural details\n\nMistral-7B is a decoder-only Transformer with the following architectural choices:\n\n- Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens\n- GQA (Grouped Query Attention) - allowing faster inference and lower cache size.\n- Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.\n\nFor more details refer to the [release blog post](https://mistral.ai/news/announcing-mistral-7b/).\n\n### License\n\n`Mistral-7B` is released under the Apache 2.0 license.\n\n## Usage tips\n\nThe Mistral team has released 3 checkpoints:\n\n- a base model, [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1), which has been pre-trained to predict the next token on internet-scale data.\n- an instruction tuned model, [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), which is the base model optimized for chat purposes using supervised fine-tuning (SFT) and direct preference optimization (DPO).\n- an improved instruction tuned model, [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), which improves upon v1."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1a78d92a-948b-4beb-9941-1cb9c60dadc7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1a78d92a-948b-4beb-9941-1cb9c60dadc7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97fee460-6c8e-4982-bc78-2b437b62f5ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6f00947c-bb43-415e-a9cb-8da1ed3424a3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ccad838274504e80eeb8c72d6d980aa1e0dec9991647f34d432f1eeb45c5197e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"110fa4b2-0809-4784-8338-2478a13fa983\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"acf4615c01abc181125911bd2f6ab64d9fc43d6e933825e54f032dd7ad2f5bcb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 438, \"end_char_idx\": 3176, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "ref_doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "chroma:document": "The introduction of the blog post says:\n\n*Mistral AI team is proud to release Mistral 7B, the most powerful language model for its size to date.*\n\nMistral-7B is the first large language model (LLM) released by [mistral.ai](https://mistral.ai/).\n\n### Architectural details\n\nMistral-7B is a decoder-only Transformer with the following architectural choices:\n\n- Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens\n- GQA (Grouped Query Attention) - allowing faster inference and lower cache size.\n- Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.\n\nFor more details refer to the [release blog post](https://mistral.ai/news/announcing-mistral-7b/).\n\n### License\n\n`Mistral-7B` is released under the Apache 2.0 license.\n\n## Usage tips\n\nThe Mistral team has released 3 checkpoints:\n\n- a base model, [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1), which has been pre-trained to predict the next token on internet-scale data.\n- an instruction tuned model, [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), which is the base model optimized for chat purposes using supervised fine-tuning (SFT) and direct preference optimization (DPO).\n- an improved instruction tuned model, [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), which improves upon v1.\n\nThe base model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"My favourite condiment is to ...\"\n```\n\nThe instruction tuned model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\")\n\n>>> messages = [\n...     {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 110fa4b2-0809-4784-8338-2478a13fa983
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"110fa4b2-0809-4784-8338-2478a13fa983\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97fee460-6c8e-4982-bc78-2b437b62f5ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1a78d92a-948b-4beb-9941-1cb9c60dadc7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"861b71bdaa695c421eb82711931ab8b2f4610ca1caa2cb7d54d74ae6b23e6e26\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ce67e6dd-92eb-4748-97a4-5f2ab8b72667\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f398234579368a2efad416727f2e3c29e45e8a0cf6fa257ed517f1d4d332f95\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1749, \"end_char_idx\": 4402, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "ref_doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "chroma:document": "- an improved instruction tuned model, [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), which improves upon v1.\n\nThe base model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"My favourite condiment is to ...\"\n```\n\nThe instruction tuned model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\")\n\n>>> messages = [\n...     {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"Mayonnaise can be made as follows: (...)\"\n```\n\nAs can be seen, the instruction-tuned model requires a [chat template](../chat_templating) to be applied to make sure the inputs are prepared in the right format.\n\n## Speeding up Mistral by using Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ce67e6dd-92eb-4748-97a4-5f2ab8b72667
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ce67e6dd-92eb-4748-97a4-5f2ab8b72667\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97fee460-6c8e-4982-bc78-2b437b62f5ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"110fa4b2-0809-4784-8338-2478a13fa983\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"31ed21d3e5ebb6607b889c0637e7f3c30580066a308a68838f79a15ae3275701\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"49cc85e1-8ccd-4390-8633-3f8cf4cb0634\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"46b909751e71ee9ddaf861cae077377f179f11f6fcf07f9b53ffe2866913b26b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2903, \"end_char_idx\": 6227, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "ref_doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "chroma:document": "\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"Mayonnaise can be made as follows: (...)\"\n```\n\nAs can be seen, the instruction-tuned model requires a [chat template](../chat_templating) to be applied to make sure the inputs are prepared in the right format.\n\n## Speeding up Mistral by using Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\nTo load and run a model using Flash Attention-2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"My favourite condiment is to (...)\"\n```\n\n### Expected speedups\n\nBelow is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `mistralai/Mistral-7B-v0.1` checkpoint and the Flash Attention 2 version of the model.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/mistral-7b-inference-large-seqlen.png\">\n</div>\n\n### Sliding window Attention\n\nThe current implementation supports the sliding window attention mechanism and memory efficient cache management. \nTo enable sliding window attention, just make sure to have a `flash-attn` version that is compatible with sliding window attention (`>=2.3.0`). \n\nThe Flash Attention-2 model uses also a more memory efficient cache slicing mechanism - as recommended per the official implementation of Mistral model that use rolling cache mechanism we keep the cache size fixed (`self.config.sliding_window`), support batched generation only for `padding_side=\"left\"` and use the absolute position of the current token to compute the positional embedding."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 49cc85e1-8ccd-4390-8633-3f8cf4cb0634
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"49cc85e1-8ccd-4390-8633-3f8cf4cb0634\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97fee460-6c8e-4982-bc78-2b437b62f5ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ce67e6dd-92eb-4748-97a4-5f2ab8b72667\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba58db1a70d3158c3c9cbe25f48c8b48bc301045762e59b44358aae7bf5c422d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"129110b6-61b3-4897-ab75-7495892f727c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4741cf3d0aeffb72a425d375f5a6d9392811808703d34a151d6cbfdafaf5adc4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5379, \"end_char_idx\": 8620, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "ref_doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "chroma:document": "<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/mistral-7b-inference-large-seqlen.png\">\n</div>\n\n### Sliding window Attention\n\nThe current implementation supports the sliding window attention mechanism and memory efficient cache management. \nTo enable sliding window attention, just make sure to have a `flash-attn` version that is compatible with sliding window attention (`>=2.3.0`). \n\nThe Flash Attention-2 model uses also a more memory efficient cache slicing mechanism - as recommended per the official implementation of Mistral model that use rolling cache mechanism we keep the cache size fixed (`self.config.sliding_window`), support batched generation only for `padding_side=\"left\"` and use the absolute position of the current token to compute the positional embedding.\n\n## Shrinking down Mistral using quantization\n\nAs the Mistral model has 7 billion parameters, that would require about 14GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter),that requires only about 3.5GB of RAM.\n\nQuantizing a model is as simple as passing a `quantization_config` to the model. Below, we'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\n>>> # specify how to quantize the model\n>>> quantization_config = BitsAndBytesConfig(\n...         load_in_4bit=True,\n...         bnb_4bit_quant_type=\"nf4\",\n...         bnb_4bit_compute_dtype=\"torch.float16\",\n... )\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\", quantization_config=True, device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> messages = [\n...     {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"The expected output\"\n```\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .\nThe original code can be found [here](https://github.com/mistralai/mistral-src).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mistral. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 129110b6-61b3-4897-ab75-7495892f727c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"129110b6-61b3-4897-ab75-7495892f727c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97fee460-6c8e-4982-bc78-2b437b62f5ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"49cc85e1-8ccd-4390-8633-3f8cf4cb0634\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8a0fffdba35c07f00f0d33f4de8075b576a6b03b02cfd2277d4d1d5ec4c3260b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7523, \"end_char_idx\": 10155, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "ref_doc_id": "97fee460-6c8e-4982-bc78-2b437b62f5ad", "chroma:document": "\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"The expected output\"\n```\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .\nThe original code can be found [here](https://github.com/mistralai/mistral-src).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mistral. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A demo notebook to perform supervised fine-tuning (SFT) of Mistral-7B can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Mistral/Supervised_fine_tuning_(SFT)_of_an_LLM_using_Hugging_Face_tooling.ipynb). \ud83c\udf0e\n- A [blog post](https://www.philschmid.de/fine-tune-llms-in-2024-with-trl) on how to fine-tune LLMs in 2024 using Hugging Face tooling. \ud83c\udf0e\n- The [Alignment Handbook](https://github.com/huggingface/alignment-handbook) by Hugging Face includes scripts and recipes to perform supervised fine-tuning (SFT) and direct preference optimization with Mistral-7B. This includes scripts for full fine-tuning, QLoRa on a single GPU as well as multi-GPU fine-tuning.\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## MistralConfig\n\n[[autodoc]] MistralConfig\n\n## MistralModel\n\n[[autodoc]] MistralModel\n    - forward\n\n## MistralForCausalLM\n\n[[autodoc]] MistralForCausalLM\n    - forward\n\n## MistralForSequenceClassification\n\n[[autodoc]] MistralForSequenceClassification\n    - forward\n\n## MistralForTokenClassification\n\n[[autodoc]] MistralForTokenClassification\n    - forward\n\n## FlaxMistralModel\n\n[[autodoc]] FlaxMistralModel\n    - __call__\n\n## FlaxMistralForCausalLM\n\n[[autodoc]] FlaxMistralForCausalLM\n    - __call__\n\n## TFMistralModel\n\n[[autodoc]] TFMistralModel\n    - call\n\n## TFMistralForCausalLM\n\n[[autodoc]] TFMistralForCausalLM\n    - call\n\n## TFMistralForSequenceClassification\n\n[[autodoc]] TFMistralForSequenceClassification\n    - call"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1ee56e15-0e0c-4919-afa5-05d1d3061dcf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swin2sr", "title": "Swin2SR", "tokens": 850, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1ee56e15-0e0c-4919-afa5-05d1d3061dcf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin2sr\", \"title\": \"Swin2SR\", \"tokens\": 850, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"db322260-e877-40b6-88fc-31d055336130\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin2sr\", \"title\": \"Swin2SR\", \"tokens\": 850, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"343ec08353c1cb319d97cb30e003eac6d53690b2e306ee1d0e9ed6062b948745\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2878, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "db322260-e877-40b6-88fc-31d055336130", "doc_id": "db322260-e877-40b6-88fc-31d055336130", "ref_doc_id": "db322260-e877-40b6-88fc-31d055336130", "chroma:document": "# Swin2SR\n\n## Overview\n\nThe Swin2SR model was proposed in [Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration](https://arxiv.org/abs/2209.11345) by Marcos V. Conde, Ui-Jin Choi, Maxime Burchi, Radu Timofte.\nSwin2R improves the [SwinIR](https://github.com/JingyunLiang/SwinIR/) model by incorporating [Swin Transformer v2](swinv2) layers which mitigates issues such as training instability, resolution gaps between pre-training\nand fine-tuning, and hunger on data.\n\nThe abstract from the paper is the following:\n\n*Compression plays an important role on the efficient transmission and storage of images and videos through band-limited systems such as streaming services, virtual reality or videogames. However, compression unavoidably leads to artifacts and the loss of the original information, which may severely degrade the visual quality. For these reasons, quality enhancement of compressed images has become a popular research topic. While most state-of-the-art image restoration methods are based on convolutional neural networks, other transformers-based methods such as SwinIR, show impressive performance on these tasks.\nIn this paper, we explore the novel Swin Transformer V2, to improve SwinIR for image super-resolution, and in particular, the compressed input scenario. Using this method we can tackle the major issues in training transformer vision models, such as training instability, resolution gaps between pre-training and fine-tuning, and hunger on data. We conduct experiments on three representative tasks: JPEG compression artifacts removal, image super-resolution (classical and lightweight), and compressed image super-resolution. Experimental results demonstrate that our method, Swin2SR, can improve the training convergence and performance of SwinIR, and is a top-5 solution at the \"AIM 2022 Challenge on Super-Resolution of Compressed Image and Video\".*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/swin2sr_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Swin2SR architecture. Taken from the <a href=\"https://arxiv.org/abs/2209.11345\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/mv-lab/swin2sr).\n\n## Resources\n\nDemo notebooks for Swin2SR can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Swin2SR).\n\nA demo Space for image super-resolution with SwinSR can be found [here](https://huggingface.co/spaces/jjourney1125/swin2sr).\n\n## Swin2SRImageProcessor\n\n[[autodoc]] Swin2SRImageProcessor\n    - preprocess\n\n## Swin2SRConfig\n\n[[autodoc]] Swin2SRConfig\n\n## Swin2SRModel\n\n[[autodoc]] Swin2SRModel\n    - forward\n\n## Swin2SRForImageSuperResolution\n\n[[autodoc]] Swin2SRForImageSuperResolution\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5963cbd6-faf4-4998-b693-c04485c956be
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flava", "title": "FLAVA", "tokens": 777, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5963cbd6-faf4-4998-b693-c04485c956be\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flava\", \"title\": \"FLAVA\", \"tokens\": 777, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e9467f1-a698-478b-a50c-636abb40ce22\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flava\", \"title\": \"FLAVA\", \"tokens\": 777, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"50336de5b042fc0e07c287fe075c94818fd039cb0ecf2fd39b0e4cc488b2f61d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2373, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e9467f1-a698-478b-a50c-636abb40ce22", "doc_id": "4e9467f1-a698-478b-a50c-636abb40ce22", "ref_doc_id": "4e9467f1-a698-478b-a50c-636abb40ce22", "chroma:document": "# FLAVA\n\n## Overview\n\nThe FLAVA model was proposed in [FLAVA: A Foundational Language And Vision Alignment Model](https://arxiv.org/abs/2112.04482) by Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela and is accepted at CVPR 2022.\n\nThe paper aims at creating a single unified foundation model which can work across vision, language\nas well as vision-and-language multimodal tasks.\n\nThe abstract from the paper is the following:\n\n*State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety\nof downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal\n(with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising\ndirection would be to use a single holistic universal model, as a \"foundation\", that targets all modalities\nat once -- a true vision and language foundation model should be good at vision tasks, language tasks, and\ncross- and multi-modal vision and language tasks. We introduce FLAVA as such a model and demonstrate\nimpressive performance on a wide range of 35 tasks spanning these target modalities.*\n\nThis model was contributed by [aps](https://huggingface.co/aps). The original code can be found [here](https://github.com/facebookresearch/multimodal/tree/main/examples/flava).\n\n## FlavaConfig\n\n[[autodoc]] FlavaConfig\n\n## FlavaTextConfig\n\n[[autodoc]] FlavaTextConfig\n\n## FlavaImageConfig\n\n[[autodoc]] FlavaImageConfig\n\n## FlavaMultimodalConfig\n\n[[autodoc]] FlavaMultimodalConfig\n\n## FlavaImageCodebookConfig\n\n[[autodoc]] FlavaImageCodebookConfig\n\n## FlavaProcessor\n\n[[autodoc]] FlavaProcessor\n\n## FlavaFeatureExtractor\n\n[[autodoc]] FlavaFeatureExtractor\n\n## FlavaImageProcessor\n\n[[autodoc]] FlavaImageProcessor\n    - preprocess\n\n## FlavaForPreTraining\n\n[[autodoc]] FlavaForPreTraining\n    - forward\n\n## FlavaModel\n\n[[autodoc]] FlavaModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## FlavaImageCodebook\n\n[[autodoc]] FlavaImageCodebook\n    - forward\n    - get_codebook_indices\n    - get_codebook_probs\n\n## FlavaTextModel\n\n[[autodoc]] FlavaTextModel\n    - forward\n\n## FlavaImageModel\n\n[[autodoc]] FlavaImageModel\n    - forward\n\n## FlavaMultimodalModel\n\n[[autodoc]] FlavaMultimodalModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c207d720-55db-43d7-85ff-21c5a7006d1d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinat", "title": "Dilated Neighborhood Attention Transformer", "tokens": 1172, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c207d720-55db-43d7-85ff-21c5a7006d1d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e50b17be-2bab-4fe2-bc3f-fb4341a29b05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a1072cbe1f2b645db94485c34c83e539ad487047e286b4cc53bdadc1154c23f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"928a0624-8275-4052-bff2-5ab6023f38df\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fcd5672203e28fa6856a2a6a1d30c5a639f8e497b01722aa4528c3edffb35d13\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3415, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e50b17be-2bab-4fe2-bc3f-fb4341a29b05", "doc_id": "e50b17be-2bab-4fe2-bc3f-fb4341a29b05", "ref_doc_id": "e50b17be-2bab-4fe2-bc3f-fb4341a29b05", "chroma:document": "# Dilated Neighborhood Attention Transformer\n\n## Overview\n\nDiNAT was proposed in [Dilated Neighborhood Attention Transformer](https://arxiv.org/abs/2209.15001)\nby Ali Hassani and Humphrey Shi.\n\nIt extends [NAT](nat) by adding a Dilated Neighborhood Attention pattern to capture global context,\nand shows significant performance improvements over it.\n\nThe abstract from the paper is the following:\n\n*Transformers are quickly becoming one of the most heavily applied deep learning architectures across modalities,\ndomains, and tasks. In vision, on top of ongoing efforts into plain transformers, hierarchical transformers have\nalso gained significant attention, thanks to their performance and easy integration into existing frameworks.\nThese models typically employ localized attention mechanisms, such as the sliding-window Neighborhood Attention (NA)\nor Swin Transformer's Shifted Window Self Attention. While effective at reducing self attention's quadratic complexity,\nlocal attention weakens two of the most desirable properties of self attention: long range inter-dependency modeling,\nand global receptive field. In this paper, we introduce Dilated Neighborhood Attention (DiNA), a natural, flexible and\nefficient extension to NA that can capture more global context and expand receptive fields exponentially at no\nadditional cost. NA's local attention and DiNA's sparse global attention complement each other, and therefore we\nintroduce Dilated Neighborhood Attention Transformer (DiNAT), a new hierarchical vision transformer built upon both.\nDiNAT variants enjoy significant improvements over strong baselines such as NAT, Swin, and ConvNeXt.\nOur large model is faster and ahead of its Swin counterpart by 1.5% box AP in COCO object detection,\n1.3% mask AP in COCO instance segmentation, and 1.1% mIoU in ADE20K semantic segmentation.\nPaired with new frameworks, our large variant is the new state of the art panoptic segmentation model on COCO (58.2 PQ)\nand ADE20K (48.5 PQ), and instance segmentation model on Cityscapes (44.5 AP) and ADE20K (35.4 AP) (no extra data).\nIt also matches the state of the art specialized semantic segmentation models on ADE20K (58.2 mIoU),\nand ranks second on Cityscapes (84.5 mIoU) (no extra data). *\n\n<img\nsrc=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dilated-neighborhood-attention-pattern.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Neighborhood Attention with different dilation values.\nTaken from the <a href=\"https://arxiv.org/abs/2209.15001\">original paper</a>.</small>\n\nThis model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).\nThe original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).\n\n## Usage tips\n\nDiNAT can be used as a *backbone*. When `output_hidden_states = True`,\nit will output both `hidden_states` and `reshaped_hidden_states`. The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than `(batch_size, height, width, num_channels)`.\n\nNotes:\n- DiNAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention and Dilated Neighborhood Attention.\nYou can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten), or build on your system by running `pip install natten`.\nNote that the latter will likely take time to compile."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 928a0624-8275-4052-bff2-5ab6023f38df
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinat", "title": "Dilated Neighborhood Attention Transformer", "tokens": 1172, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"928a0624-8275-4052-bff2-5ab6023f38df\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e50b17be-2bab-4fe2-bc3f-fb4341a29b05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a1072cbe1f2b645db94485c34c83e539ad487047e286b4cc53bdadc1154c23f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c207d720-55db-43d7-85ff-21c5a7006d1d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a2bc34ee053c07572f21fd2ad1a957e74d71c53ae44c4f6502c86587768e7102\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2079, \"end_char_idx\": 4461, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e50b17be-2bab-4fe2-bc3f-fb4341a29b05", "doc_id": "e50b17be-2bab-4fe2-bc3f-fb4341a29b05", "ref_doc_id": "e50b17be-2bab-4fe2-bc3f-fb4341a29b05", "chroma:document": "It also matches the state of the art specialized semantic segmentation models on ADE20K (58.2 mIoU),\nand ranks second on Cityscapes (84.5 mIoU) (no extra data). *\n\n<img\nsrc=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dilated-neighborhood-attention-pattern.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Neighborhood Attention with different dilation values.\nTaken from the <a href=\"https://arxiv.org/abs/2209.15001\">original paper</a>.</small>\n\nThis model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).\nThe original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).\n\n## Usage tips\n\nDiNAT can be used as a *backbone*. When `output_hidden_states = True`,\nit will output both `hidden_states` and `reshaped_hidden_states`. The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than `(batch_size, height, width, num_channels)`.\n\nNotes:\n- DiNAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention and Dilated Neighborhood Attention.\nYou can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten), or build on your system by running `pip install natten`.\nNote that the latter will likely take time to compile. NATTEN does not support Windows devices yet.\n- Patch size of 4 is only supported at the moment.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DiNAT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`DinatForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DinatConfig\n\n[[autodoc]] DinatConfig\n\n## DinatModel\n\n[[autodoc]] DinatModel\n    - forward\n\n## DinatForImageClassification\n\n[[autodoc]] DinatForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e1acff50-7676-4a98-9d75-032dd72e7041
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-conformer", "title": "Wav2Vec2-Conformer", "tokens": 922, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e1acff50-7676-4a98-9d75-032dd72e7041\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-conformer\", \"title\": \"Wav2Vec2-Conformer\", \"tokens\": 922, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2ac93837-11dd-4aea-8ac0-2af8480f7e6a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-conformer\", \"title\": \"Wav2Vec2-Conformer\", \"tokens\": 922, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d265c77f0218a4c660aeb6d8597bdb066e5a6a889536ec3a83dc80ee41d44c68\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2653, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2ac93837-11dd-4aea-8ac0-2af8480f7e6a", "doc_id": "2ac93837-11dd-4aea-8ac0-2af8480f7e6a", "ref_doc_id": "2ac93837-11dd-4aea-8ac0-2af8480f7e6a", "chroma:document": "# Wav2Vec2-Conformer\n\n## Overview\n\nThe Wav2Vec2-Conformer was added to an updated version of [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://arxiv.org/abs/2010.05171) by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Sravya Popuri, Dmytro Okhonko, Juan Pino.\n\nThe official results of the model can be found in Table 3 and Table 4 of the paper.\n\nThe Wav2Vec2-Conformer weights were released by the Meta AI team within the [Fairseq library](https://github.com/pytorch/fairseq/blob/main/examples/wav2vec/README.md#pre-trained-models).\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).\nThe original code can be found [here](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec).\n\nNote: Meta (FAIR) released a new version of [Wav2Vec2-BERT 2.0](https://huggingface.co/docs/transformers/en/model_doc/wav2vec2-bert) - it's pretrained on 4.5M hours of audio. We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).\n\n## Usage tips\n\n- Wav2Vec2-Conformer follows the same architecture as Wav2Vec2, but replaces the *Attention*-block with a *Conformer*-block\n  as introduced in [Conformer: Convolution-augmented Transformer for Speech Recognition](https://arxiv.org/abs/2005.08100).\n- For the same number of layers, Wav2Vec2-Conformer requires more parameters than Wav2Vec2, but also yields \nan improved word error rate.\n- Wav2Vec2-Conformer uses the same tokenizer and feature extractor as Wav2Vec2.\n- Wav2Vec2-Conformer can use either no relative position embeddings, Transformer-XL-like position embeddings, or\n  rotary position embeddings by setting the correct `config.position_embeddings_type`.\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## Wav2Vec2ConformerConfig\n\n[[autodoc]] Wav2Vec2ConformerConfig\n\n## Wav2Vec2Conformer specific outputs\n\n[[autodoc]] models.wav2vec2_conformer.modeling_wav2vec2_conformer.Wav2Vec2ConformerForPreTrainingOutput\n\n## Wav2Vec2ConformerModel\n\n[[autodoc]] Wav2Vec2ConformerModel\n    - forward\n\n## Wav2Vec2ConformerForCTC\n\n[[autodoc]] Wav2Vec2ConformerForCTC\n    - forward\n\n## Wav2Vec2ConformerForSequenceClassification\n\n[[autodoc]] Wav2Vec2ConformerForSequenceClassification\n    - forward\n\n## Wav2Vec2ConformerForAudioFrameClassification\n\n[[autodoc]] Wav2Vec2ConformerForAudioFrameClassification\n    - forward\n\n## Wav2Vec2ConformerForXVector\n\n[[autodoc]] Wav2Vec2ConformerForXVector\n    - forward\n\n## Wav2Vec2ConformerForPreTraining\n\n[[autodoc]] Wav2Vec2ConformerForPreTraining\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7225e4db-0b0e-4fd2-afc1-7ee7c42202f3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7225e4db-0b0e-4fd2-afc1-7ee7c42202f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba797f60-f9b4-407c-8260-752741094b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ce9405a-5c16-41a4-aad0-3a73cced18cd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2f19aa9b5a565347a92a30d9309605ab341a4bb34e1004d3d4acab511403a1d8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3308, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba797f60-f9b4-407c-8260-752741094b6c", "doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "ref_doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "chroma:document": "# MarkupLM\n\n## Overview\n\nThe MarkupLM model was proposed in [MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document\nUnderstanding](https://arxiv.org/abs/2110.08518) by Junlong Li, Yiheng Xu, Lei Cui, Furu Wei. MarkupLM is BERT, but\napplied to HTML pages instead of raw text documents. The model incorporates additional embedding layers to improve\nperformance, similar to [LayoutLM](layoutlm).\n\nThe model can be used for tasks like question answering on web pages or information extraction from web pages. It obtains\nstate-of-the-art results on 2 important benchmarks:\n- [WebSRC](https://x-lance.github.io/WebSRC/), a dataset for Web-Based Structural Reading Comprehension (a bit like SQuAD but for web pages)\n- [SWDE](https://www.researchgate.net/publication/221299838_From_one_tree_to_a_forest_a_unified_solution_for_structured_web_data_extraction), a dataset\nfor information extraction from web pages (basically named-entity recognition on web pages)\n\nThe abstract from the paper is the following:\n\n*Multimodal pre-training with text, layout, and image has made significant progress for Visually-rich Document\nUnderstanding (VrDU), especially the fixed-layout documents such as scanned document images. While, there are still a\nlarge number of digital documents where the layout information is not fixed and needs to be interactively and\ndynamically rendered for visualization, making existing layout-based pre-training approaches not easy to apply. In this\npaper, we propose MarkupLM for document understanding tasks with markup languages as the backbone such as\nHTML/XML-based documents, where text and markup information is jointly pre-trained. Experiment results show that the\npre-trained MarkupLM significantly outperforms the existing strong baseline models on several document understanding\ntasks. The pre-trained model and code will be publicly available.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/markuplm).\n\n## Usage tips\n\n- In addition to `input_ids`, [`~MarkupLMModel.forward`] expects 2 additional inputs, namely `xpath_tags_seq` and `xpath_subs_seq`.\nThese are the XPATH tags and subscripts respectively for each token in the input sequence.\n- One can use [`MarkupLMProcessor`] to prepare all data for the model. Refer to the [usage guide](#usage-markuplmprocessor) for more info.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/markuplm_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> MarkupLM architecture. Taken from the <a href=\"https://arxiv.org/abs/2110.08518\">original paper.</a> </small>\n\n## Usage: MarkupLMProcessor\n\nThe easiest way to prepare data for the model is to use [`MarkupLMProcessor`], which internally combines a feature extractor\n([`MarkupLMFeatureExtractor`]) and a tokenizer ([`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`]). The feature extractor is\nused to extract all nodes and xpaths from the HTML strings, which are then provided to the tokenizer, which turns them into the\ntoken-level inputs of the model (`input_ids` etc.). Note that you can still use the feature extractor and tokenizer separately,\nif you only want to handle one of the two tasks."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4ce9405a-5c16-41a4-aad0-3a73cced18cd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4ce9405a-5c16-41a4-aad0-3a73cced18cd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba797f60-f9b4-407c-8260-752741094b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7225e4db-0b0e-4fd2-afc1-7ee7c42202f3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e65c5a58947af6692272e8de14f652cac694725f2a1a2ea8c1e2c4b65927753a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"95b22bb0-8f05-43e8-81bd-5395a0aea5d3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5c81aac119072dcaf171a4d4a7cc2bc350f20bfd72f3dd1e9b4b4d9af9d25b3f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1673, \"end_char_idx\": 4990, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba797f60-f9b4-407c-8260-752741094b6c", "doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "ref_doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "chroma:document": "Experiment results show that the\npre-trained MarkupLM significantly outperforms the existing strong baseline models on several document understanding\ntasks. The pre-trained model and code will be publicly available.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/markuplm).\n\n## Usage tips\n\n- In addition to `input_ids`, [`~MarkupLMModel.forward`] expects 2 additional inputs, namely `xpath_tags_seq` and `xpath_subs_seq`.\nThese are the XPATH tags and subscripts respectively for each token in the input sequence.\n- One can use [`MarkupLMProcessor`] to prepare all data for the model. Refer to the [usage guide](#usage-markuplmprocessor) for more info.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/markuplm_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> MarkupLM architecture. Taken from the <a href=\"https://arxiv.org/abs/2110.08518\">original paper.</a> </small>\n\n## Usage: MarkupLMProcessor\n\nThe easiest way to prepare data for the model is to use [`MarkupLMProcessor`], which internally combines a feature extractor\n([`MarkupLMFeatureExtractor`]) and a tokenizer ([`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`]). The feature extractor is\nused to extract all nodes and xpaths from the HTML strings, which are then provided to the tokenizer, which turns them into the\ntoken-level inputs of the model (`input_ids` etc.). Note that you can still use the feature extractor and tokenizer separately,\nif you only want to handle one of the two tasks.\n\n```python\nfrom transformers import MarkupLMFeatureExtractor, MarkupLMTokenizerFast, MarkupLMProcessor\n\nfeature_extractor = MarkupLMFeatureExtractor()\ntokenizer = MarkupLMTokenizerFast.from_pretrained(\"microsoft/markuplm-base\")\nprocessor = MarkupLMProcessor(feature_extractor, tokenizer)\n```\n\nIn short, one can provide HTML strings (and possibly additional data) to [`MarkupLMProcessor`],\nand it will create the inputs expected by the model. Internally, the processor first uses\n[`MarkupLMFeatureExtractor`] to get a list of nodes and corresponding xpaths. The nodes and\nxpaths are then provided to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`], which converts them\nto token-level `input_ids`, `attention_mask`, `token_type_ids`, `xpath_subs_seq`, `xpath_tags_seq`.\nOptionally, one can provide node labels to the processor, which are turned into token-level `labels`.\n\n[`MarkupLMFeatureExtractor`] uses [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/), a Python library for\npulling data out of HTML and XML files, under the hood. Note that you can still use your own parsing solution of\nchoice, and provide the nodes and xpaths yourself to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`].\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: web page classification (training, inference) + token classification (inference), parse_html = True**\n\nThis is the simplest case, in which the processor will use the feature extractor to get all nodes and xpaths from the HTML."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 95b22bb0-8f05-43e8-81bd-5395a0aea5d3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"95b22bb0-8f05-43e8-81bd-5395a0aea5d3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba797f60-f9b4-407c-8260-752741094b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4ce9405a-5c16-41a4-aad0-3a73cced18cd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7e9d808b4a5e9fe746816e01ff850cb61a845b24397bcf49221a3b490cabb322\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8e1699a5-98e8-4844-b449-71af94a1c695\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"10325888c18faf8589a4ab989b9d9c46832dd55801ce969ae171ac90b060144d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3310, \"end_char_idx\": 5976, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba797f60-f9b4-407c-8260-752741094b6c", "doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "ref_doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "chroma:document": "```python\nfrom transformers import MarkupLMFeatureExtractor, MarkupLMTokenizerFast, MarkupLMProcessor\n\nfeature_extractor = MarkupLMFeatureExtractor()\ntokenizer = MarkupLMTokenizerFast.from_pretrained(\"microsoft/markuplm-base\")\nprocessor = MarkupLMProcessor(feature_extractor, tokenizer)\n```\n\nIn short, one can provide HTML strings (and possibly additional data) to [`MarkupLMProcessor`],\nand it will create the inputs expected by the model. Internally, the processor first uses\n[`MarkupLMFeatureExtractor`] to get a list of nodes and corresponding xpaths. The nodes and\nxpaths are then provided to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`], which converts them\nto token-level `input_ids`, `attention_mask`, `token_type_ids`, `xpath_subs_seq`, `xpath_tags_seq`.\nOptionally, one can provide node labels to the processor, which are turned into token-level `labels`.\n\n[`MarkupLMFeatureExtractor`] uses [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/), a Python library for\npulling data out of HTML and XML files, under the hood. Note that you can still use your own parsing solution of\nchoice, and provide the nodes and xpaths yourself to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`].\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: web page classification (training, inference) + token classification (inference), parse_html = True**\n\nThis is the simplest case, in which the processor will use the feature extractor to get all nodes and xpaths from the HTML.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n\n>>> html_string = \"\"\"\n...  <!DOCTYPE html>\n...  <html>\n...  <head>\n...  <title>Hello world</title>\n...  </head>\n...  <body>\n...  <h1>Welcome</h1>\n...  <p>Here is my website.</p>\n...  </body>\n...  </html>\"\"\"\n\n>>> # note that you can also add provide all tokenizer parameters here such as padding, truncation\n>>> encoding = processor(html_string, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 2: web page classification (training, inference) + token classification (inference), parse_html=False**\n\nIn case one already has obtained all nodes and xpaths, one doesn't need the feature extractor. In that case, one should\nprovide the nodes and corresponding xpaths themselves to the processor, and make sure to set `parse_html` to `False`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8e1699a5-98e8-4844-b449-71af94a1c695
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8e1699a5-98e8-4844-b449-71af94a1c695\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba797f60-f9b4-407c-8260-752741094b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"95b22bb0-8f05-43e8-81bd-5395a0aea5d3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e1471c3083f5d3bde961e8de6da27fb421d6b915c03802c2da511d3eff0c354d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"968b527e-1cad-427e-b036-7a56303d0486\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"831c2b10a3bfaf1647354f7005aca8c680dd078f16e418a3a45ae510cb85e431\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4368, \"end_char_idx\": 7109, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba797f60-f9b4-407c-8260-752741094b6c", "doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "ref_doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "chroma:document": "Note that you can still use your own parsing solution of\nchoice, and provide the nodes and xpaths yourself to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`].\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: web page classification (training, inference) + token classification (inference), parse_html = True**\n\nThis is the simplest case, in which the processor will use the feature extractor to get all nodes and xpaths from the HTML.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n\n>>> html_string = \"\"\"\n...  <!DOCTYPE html>\n...  <html>\n...  <head>\n...  <title>Hello world</title>\n...  </head>\n...  <body>\n...  <h1>Welcome</h1>\n...  <p>Here is my website.</p>\n...  </body>\n...  </html>\"\"\"\n\n>>> # note that you can also add provide all tokenizer parameters here such as padding, truncation\n>>> encoding = processor(html_string, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 2: web page classification (training, inference) + token classification (inference), parse_html=False**\n\nIn case one already has obtained all nodes and xpaths, one doesn't need the feature extractor. In that case, one should\nprovide the nodes and corresponding xpaths themselves to the processor, and make sure to set `parse_html` to `False`.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n>>> processor.parse_html = False\n\n>>> nodes = [\"hello\", \"world\", \"how\", \"are\"]\n>>> xpaths = [\"/html/body/div/li[1]/div/span\", \"/html/body/div/li[1]/div/span\", \"html/body\", \"html/body/div\"]\n>>> encoding = processor(nodes=nodes, xpaths=xpaths, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 3: token classification (training), parse_html=False**\n\nFor token classification tasks (such as [SWDE](https://paperswithcode.com/dataset/swde)), one can also provide the\ncorresponding node labels in order to train a model. The processor will then convert these into token-level `labels`.\nBy default, it will only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the\n`ignore_index` of PyTorch's CrossEntropyLoss. In case you want all wordpieces of a word to be labeled, you can\ninitialize the tokenizer with `only_label_first_subword` set to `False`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 968b527e-1cad-427e-b036-7a56303d0486
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"968b527e-1cad-427e-b036-7a56303d0486\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba797f60-f9b4-407c-8260-752741094b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8e1699a5-98e8-4844-b449-71af94a1c695\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2e6128931e15ae9a49d90834a094ff3245f9b4228ea78dd66690ff2551e64358\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7a52d7ad-af4f-4570-bdd9-c1743bb33ec0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"290e33ba402886c42f647e0463c70ed22fde44acddf7584bcea62ab87ac4e097\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5834, \"end_char_idx\": 8904, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba797f60-f9b4-407c-8260-752741094b6c", "doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "ref_doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "chroma:document": "In that case, one should\nprovide the nodes and corresponding xpaths themselves to the processor, and make sure to set `parse_html` to `False`.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n>>> processor.parse_html = False\n\n>>> nodes = [\"hello\", \"world\", \"how\", \"are\"]\n>>> xpaths = [\"/html/body/div/li[1]/div/span\", \"/html/body/div/li[1]/div/span\", \"html/body\", \"html/body/div\"]\n>>> encoding = processor(nodes=nodes, xpaths=xpaths, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 3: token classification (training), parse_html=False**\n\nFor token classification tasks (such as [SWDE](https://paperswithcode.com/dataset/swde)), one can also provide the\ncorresponding node labels in order to train a model. The processor will then convert these into token-level `labels`.\nBy default, it will only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the\n`ignore_index` of PyTorch's CrossEntropyLoss. In case you want all wordpieces of a word to be labeled, you can\ninitialize the tokenizer with `only_label_first_subword` set to `False`.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n>>> processor.parse_html = False\n\n>>> nodes = [\"hello\", \"world\", \"how\", \"are\"]\n>>> xpaths = [\"/html/body/div/li[1]/div/span\", \"/html/body/div/li[1]/div/span\", \"html/body\", \"html/body/div\"]\n>>> node_labels = [1, 2, 2, 1]\n>>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq', 'labels'])\n```\n\n**Use case 4: web page question answering (inference), parse_html=True**\n\nFor question answering tasks on web pages, you can provide a question to the processor. By default, the\nprocessor will use the feature extractor to get all nodes and xpaths, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n\n>>> html_string = \"\"\"\n...  <!DOCTYPE html>\n...  <html>\n...  <head>\n...  <title>Hello world</title>\n...  </head>\n...  <body>\n...  <h1>Welcome</h1>\n...  <p>My name is Niels.</p>\n...  </body>\n...  </html>\"\"\"\n\n>>> question = \"What's his name?\"\n>>> encoding = processor(html_string, questions=question, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 5: web page question answering (inference), parse_html=False**\n\nFor question answering tasks (such as WebSRC), you can provide a question to the processor. If you have extracted\nall nodes and xpaths yourself, you can provide them directly to the processor. Make sure to set `parse_html` to `False`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7a52d7ad-af4f-4570-bdd9-c1743bb33ec0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7a52d7ad-af4f-4570-bdd9-c1743bb33ec0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba797f60-f9b4-407c-8260-752741094b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"968b527e-1cad-427e-b036-7a56303d0486\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1aaeaed103b7a8bf69e1de3effa3cb4c785e28dad0322427a259417cde01dfdf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7866, \"end_char_idx\": 10598, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba797f60-f9b4-407c-8260-752741094b6c", "doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "ref_doc_id": "ba797f60-f9b4-407c-8260-752741094b6c", "chroma:document": "By default, the\nprocessor will use the feature extractor to get all nodes and xpaths, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n\n>>> html_string = \"\"\"\n...  <!DOCTYPE html>\n...  <html>\n...  <head>\n...  <title>Hello world</title>\n...  </head>\n...  <body>\n...  <h1>Welcome</h1>\n...  <p>My name is Niels.</p>\n...  </body>\n...  </html>\"\"\"\n\n>>> question = \"What's his name?\"\n>>> encoding = processor(html_string, questions=question, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 5: web page question answering (inference), parse_html=False**\n\nFor question answering tasks (such as WebSRC), you can provide a question to the processor. If you have extracted\nall nodes and xpaths yourself, you can provide them directly to the processor. Make sure to set `parse_html` to `False`.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n>>> processor.parse_html = False\n\n>>> nodes = [\"hello\", \"world\", \"how\", \"are\"]\n>>> xpaths = [\"/html/body/div/li[1]/div/span\", \"/html/body/div/li[1]/div/span\", \"html/body\", \"html/body/div\"]\n>>> question = \"What's his name?\"\n>>> encoding = processor(nodes=nodes, xpaths=xpaths, questions=question, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n## Resources\n\n- [Demo notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/MarkupLM)\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n\n## MarkupLMConfig\n\n[[autodoc]] MarkupLMConfig\n    - all\n\n## MarkupLMFeatureExtractor\n\n[[autodoc]] MarkupLMFeatureExtractor\n    - __call__\n\n## MarkupLMTokenizer\n\n[[autodoc]] MarkupLMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## MarkupLMTokenizerFast\n\n[[autodoc]] MarkupLMTokenizerFast\n    - all\n\n## MarkupLMProcessor\n\n[[autodoc]] MarkupLMProcessor\n    - __call__\n\n## MarkupLMModel\n\n[[autodoc]] MarkupLMModel\n    - forward\n\n## MarkupLMForSequenceClassification\n\n[[autodoc]] MarkupLMForSequenceClassification\n    - forward\n\n## MarkupLMForTokenClassification\n\n[[autodoc]] MarkupLMForTokenClassification\n    - forward\n\n## MarkupLMForQuestionAnswering\n\n[[autodoc]] MarkupLMForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 92db49bf-42a8-43d2-bbe4-1229aa42d01b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"92db49bf-42a8-43d2-bbe4-1229aa42d01b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cd47ccf3-f986-4664-8f78-cb604e270efe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a9815164-2d69-4796-ab40-04bae4f4e0a0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3e56931d1a11feb3c87f796a1f24bb508095200f765d983558270273643e5ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3067, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "doc_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "ref_doc_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "chroma:document": "# BEiT\n\n## Overview\n\nThe BEiT model was proposed in [BEiT: BERT Pre-Training of Image Transformers](https://arxiv.org/abs/2106.08254) by\nHangbo Bao, Li Dong and Furu Wei. Inspired by BERT, BEiT is the first paper that makes self-supervised pre-training of\nVision Transformers (ViTs) outperform supervised pre-training. Rather than pre-training the model to predict the class\nof an image (as done in the [original ViT paper](https://arxiv.org/abs/2010.11929)), BEiT models are pre-trained to\npredict visual tokens from the codebook of OpenAI's [DALL-E model](https://arxiv.org/abs/2102.12092) given masked\npatches.\n\nThe abstract from the paper is the following:\n\n*We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation\nfrom Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image\nmodeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image\npatches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first \"tokenize\" the original image into\nvisual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training\nobjective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we\ndirectly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder.\nExperimental results on image classification and semantic segmentation show that our model achieves competitive results\nwith previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K,\nsignificantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains\n86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The JAX/FLAX version of this model was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/beit).\n\n## Usage tips\n\n- BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They\n  outperform both the [original model (ViT)](vit) as well as [Data-efficient Image Transformers (DeiT)](deit) when fine-tuned on ImageNet-1K and CIFAR-100. You can check out demo notebooks regarding inference as well as\n  fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace\n  [`ViTFeatureExtractor`] by [`BeitImageProcessor`] and\n  [`ViTForImageClassification`] by [`BeitForImageClassification`]).\n- There's also a demo notebook available which showcases how to combine DALL-E's image tokenizer with BEiT for\n  performing masked image modeling. You can find it [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BEiT)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a9815164-2d69-4796-ab40-04bae4f4e0a0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a9815164-2d69-4796-ab40-04bae4f4e0a0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cd47ccf3-f986-4664-8f78-cb604e270efe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"92db49bf-42a8-43d2-bbe4-1229aa42d01b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7c6257c64012a687ef28534ef3f5d7472681d437bd78e4e011ad0b1b84a67a7b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e6ec7322-38b1-47a7-9ea1-0061d70f16eb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3a81ece81201e5b44a5db6112158defb294e7caedc0eab21dfaeebe5df6d13cf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1640, \"end_char_idx\": 4555, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "doc_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "ref_doc_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "chroma:document": "For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K,\nsignificantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains\n86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The JAX/FLAX version of this model was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/beit).\n\n## Usage tips\n\n- BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They\n  outperform both the [original model (ViT)](vit) as well as [Data-efficient Image Transformers (DeiT)](deit) when fine-tuned on ImageNet-1K and CIFAR-100. You can check out demo notebooks regarding inference as well as\n  fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace\n  [`ViTFeatureExtractor`] by [`BeitImageProcessor`] and\n  [`ViTForImageClassification`] by [`BeitForImageClassification`]).\n- There's also a demo notebook available which showcases how to combine DALL-E's image tokenizer with BEiT for\n  performing masked image modeling. You can find it [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BEiT).\n- As the BEiT models expect each image to be of the same size (resolution), one can use\n  [`BeitImageProcessor`] to resize (or rescale) and normalize images for the model.\n- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of\n  each checkpoint. For example, `microsoft/beit-base-patch16-224` refers to a base-sized architecture with patch\n  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=microsoft/beit).\n- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of\n  14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the\n  relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position\n  bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to\n  pre-train a model from scratch, one needs to either set the `use_relative_position_bias` or the\n  `use_relative_position_bias` attribute of [`BeitConfig`] to `True` in order to add\n  position embeddings."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e6ec7322-38b1-47a7-9ea1-0061d70f16eb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e6ec7322-38b1-47a7-9ea1-0061d70f16eb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cd47ccf3-f986-4664-8f78-cb604e270efe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a9815164-2d69-4796-ab40-04bae4f4e0a0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c6caa421c927a35787624f2129f6db7b32f04d2e0d8f9b7979b10d119556582f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7074ae17-ae43-4ac0-acfa-894ae2ee9865\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"383aea2d91e4685f986ab066d0b16b34fe375ff09ef83b09016af3dc98ca770f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3068, \"end_char_idx\": 5696, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "doc_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "ref_doc_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "chroma:document": "- As the BEiT models expect each image to be of the same size (resolution), one can use\n  [`BeitImageProcessor`] to resize (or rescale) and normalize images for the model.\n- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of\n  each checkpoint. For example, `microsoft/beit-base-patch16-224` refers to a base-sized architecture with patch\n  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=microsoft/beit).\n- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of\n  14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the\n  relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position\n  bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to\n  pre-train a model from scratch, one needs to either set the `use_relative_position_bias` or the\n  `use_relative_position_bias` attribute of [`BeitConfig`] to `True` in order to add\n  position embeddings.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/beit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> BEiT pre-training. Taken from the <a href=\"https://arxiv.org/abs/2106.08254\">original paper.</a> </small>\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BEiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n**Semantic segmentation**\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7074ae17-ae43-4ac0-acfa-894ae2ee9865
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7074ae17-ae43-4ac0-acfa-894ae2ee9865\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cd47ccf3-f986-4664-8f78-cb604e270efe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e6ec7322-38b1-47a7-9ea1-0061d70f16eb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a09b2578565fa327e02964582e2ba12d88ce22c94f5c7c22bf5f3268b0de0a47\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4004, \"end_char_idx\": 6752, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "doc_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "ref_doc_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "chroma:document": "- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the\n  relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position\n  bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to\n  pre-train a model from scratch, one needs to either set the `use_relative_position_bias` or the\n  `use_relative_position_bias` attribute of [`BeitConfig`] to `True` in order to add\n  position embeddings.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/beit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> BEiT pre-training. Taken from the <a href=\"https://arxiv.org/abs/2106.08254\">original paper.</a> </small>\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BEiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n**Semantic segmentation**\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## BEiT specific outputs\n\n[[autodoc]] models.beit.modeling_beit.BeitModelOutputWithPooling\n\n[[autodoc]] models.beit.modeling_flax_beit.FlaxBeitModelOutputWithPooling\n\n## BeitConfig\n\n[[autodoc]] BeitConfig\n\n## BeitFeatureExtractor\n\n[[autodoc]] BeitFeatureExtractor\n    - __call__\n    - post_process_semantic_segmentation\n\n## BeitImageProcessor\n\n[[autodoc]] BeitImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n<frameworkcontent>\n<pt>\n\n## BeitModel\n\n[[autodoc]] BeitModel\n    - forward\n\n## BeitForMaskedImageModeling\n\n[[autodoc]] BeitForMaskedImageModeling\n    - forward\n\n## BeitForImageClassification\n\n[[autodoc]] BeitForImageClassification\n    - forward\n\n## BeitForSemanticSegmentation\n\n[[autodoc]] BeitForSemanticSegmentation\n    - forward\n\n</pt>\n<jax>\n\n## FlaxBeitModel\n\n[[autodoc]] FlaxBeitModel\n    - __call__\n\n## FlaxBeitForMaskedImageModeling\n\n[[autodoc]] FlaxBeitForMaskedImageModeling\n    - __call__\n\n## FlaxBeitForImageClassification\n\n[[autodoc]] FlaxBeitForImageClassification\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9aa5518f-277b-42a3-b258-9715fda3dc66
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roc_bert", "title": "RoCBert", "tokens": 843, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9aa5518f-277b-42a3-b258-9715fda3dc66\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roc_bert\", \"title\": \"RoCBert\", \"tokens\": 843, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c9a17ac-536f-461d-b031-a3ab37e3f8ce\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roc_bert\", \"title\": \"RoCBert\", \"tokens\": 843, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"29a2c7b768273c705aa7bbe6a595c6d18c24302b6a97371e897724b0777a6855\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2771, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c9a17ac-536f-461d-b031-a3ab37e3f8ce", "doc_id": "2c9a17ac-536f-461d-b031-a3ab37e3f8ce", "ref_doc_id": "2c9a17ac-536f-461d-b031-a3ab37e3f8ce", "chroma:document": "# RoCBert\n\n## Overview\n\nThe RoCBert model was proposed in [RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining](https://aclanthology.org/2022.acl-long.65.pdf)  by HuiSu, WeiweiShi, XiaoyuShen, XiaoZhou, TuoJi, JiaruiFang, JieZhou.\nIt's a pretrained Chinese language model that is robust under various forms of adversarial attacks.\n\nThe abstract from the paper is the following:\n\n*Large-scale pretrained language models have achieved SOTA results on NLP tasks. However, they have been shown\nvulnerable to adversarial attacks especially for logographic languages like Chinese. In this work, we propose\nROCBERT: a pretrained Chinese Bert that is robust to various forms of adversarial attacks like word perturbation,\nsynonyms, typos, etc. It is pretrained with the contrastive learning objective which maximizes the label consistency\nunder different synthesized adversarial examples. The model takes as input multimodal information including the\nsemantic, phonetic and visual features. We show all these features are important to the model robustness since the\nattack can be performed in all the three forms. Across 5 Chinese NLU tasks, ROCBERT outperforms strong baselines under\nthree blackbox adversarial algorithms without sacrificing the performance on clean testset. It also performs the best\nin the toxic content detection task under human-made attacks.*\n\nThis model was contributed by [weiweishi](https://huggingface.co/weiweishi).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## RoCBertConfig\n\n[[autodoc]] RoCBertConfig\n    - all\n\n## RoCBertTokenizer\n\n[[autodoc]] RoCBertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## RoCBertModel\n\n[[autodoc]] RoCBertModel\n    - forward\n\n## RoCBertForPreTraining\n\n[[autodoc]] RoCBertForPreTraining\n    - forward\n\n## RoCBertForCausalLM\n\n[[autodoc]] RoCBertForCausalLM\n    - forward\n\n## RoCBertForMaskedLM\n\n[[autodoc]] RoCBertForMaskedLM\n    - forward\n\n## RoCBertForSequenceClassification\n\n[[autodoc]] transformers.RoCBertForSequenceClassification\n    - forward\n\n## RoCBertForMultipleChoice\n\n[[autodoc]] transformers.RoCBertForMultipleChoice\n    - forward\n\n## RoCBertForTokenClassification\n\n[[autodoc]] transformers.RoCBertForTokenClassification\n    - forward\n\n## RoCBertForQuestionAnswering\n\n[[autodoc]] RoCBertForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 61e17414-8f2e-44b8-a3b4-2234eb9ecc50
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swiftformer", "title": "SwiftFormer", "tokens": 774, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"61e17414-8f2e-44b8-a3b4-2234eb9ecc50\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swiftformer\", \"title\": \"SwiftFormer\", \"tokens\": 774, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5e611041-a433-45a5-8b5d-0063384d3300\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swiftformer\", \"title\": \"SwiftFormer\", \"tokens\": 774, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a5d9bc408c2943dc09dc397aa8c00bd3a5e2a5fb6e387b674d59f1a10a4a0a40\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2865, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5e611041-a433-45a5-8b5d-0063384d3300", "doc_id": "5e611041-a433-45a5-8b5d-0063384d3300", "ref_doc_id": "5e611041-a433-45a5-8b5d-0063384d3300", "chroma:document": "# SwiftFormer\n\n## Overview\n\nThe SwiftFormer model was proposed in [SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications](https://arxiv.org/abs/2303.15446) by Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan.\n\nThe SwiftFormer paper introduces a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations in the self-attention computation with linear element-wise multiplications. A series of models called 'SwiftFormer' is built based on this, which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Even their small variant achieves 78.5% top-1 ImageNet1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2\u00d7 faster compared to MobileViT-v2.\n\nThe abstract from the paper is the following:\n\n*Self-attention has become a defacto choice for capturing global context in various vision applications. However, its quadratic computational complexity with respect to image resolution limits its use in real-time applications, especially for deployment on resource-constrained mobile devices. Although hybrid approaches have been proposed to combine the advantages of convolutions and self-attention for a better speed-accuracy trade-off, the expensive matrix multiplication operations in self-attention remain a bottleneck. In this work, we introduce a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations with linear element-wise multiplications. Our design shows that the key-value interaction can be replaced with a linear layer without sacrificing any accuracy. Unlike previous state-of-the-art methods, our efficient formulation of self-attention enables its usage at all stages of the network. Using our proposed efficient additive attention, we build a series of models called \"SwiftFormer\" which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Our small variant achieves 78.5% top-1 ImageNet-1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2x faster compared to MobileViT-v2.*\n\nThis model was contributed by [shehan97](https://huggingface.co/shehan97). The TensorFlow version was contributed by [joaocmd](https://huggingface.co/joaocmd).\nThe original code can be found [here](https://github.com/Amshaker/SwiftFormer).\n\n## SwiftFormerConfig\n\n[[autodoc]] SwiftFormerConfig\n\n## SwiftFormerModel\n\n[[autodoc]] SwiftFormerModel\n    - forward\n\n## SwiftFormerForImageClassification\n\n[[autodoc]] SwiftFormerForImageClassification\n    - forward\n\n## TFSwiftFormerModel\n\n[[autodoc]] TFSwiftFormerModel\n    - call\n\n## TFSwiftFormerForImageClassification\n\n[[autodoc]] TFSwiftFormerForImageClassification\n    - call"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e9197f4c-9eb9-480b-85e1-a908d08756c9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e9197f4c-9eb9-480b-85e1-a908d08756c9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"af0bf3ef-b81a-4a32-8c6f-5ae79741c01d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b07c43ff-6602-46da-a4a3-ab5c3251164a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d9383a3826a20c50a2c23a9d66c5773e749623bac8849a6c1402d3447bbd68c1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3651, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "ref_doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "chroma:document": "# SeamlessM4T-v2\n\n## Overview\n\nThe SeamlessM4T-v2 model was proposed in [Seamless: Multilingual Expressive and Streaming Speech Translation](https://ai.meta.com/research/publications/seamless-multilingual-expressive-and-streaming-speech-translation/) by the Seamless Communication team from Meta AI.\n\nSeamlessM4T-v2 is a collection of models designed to provide high quality translation, allowing people from different linguistic communities to communicate effortlessly through speech and text. It is an improvement on the [previous version](https://huggingface.co/docs/transformers/main/model_doc/seamless_m4t). For more details on the differences between v1 and v2, refer to section [Difference with SeamlessM4T-v1](#difference-with-seamlessm4t-v1).\n\nSeamlessM4T-v2 enables multiple tasks without relying on separate models:\n\n- Speech-to-speech translation (S2ST)\n- Speech-to-text translation (S2TT)\n- Text-to-speech translation (T2ST)\n- Text-to-text translation (T2TT)\n- Automatic speech recognition (ASR)\n\n[`SeamlessM4Tv2Model`] can perform all the above tasks, but each task also has its own dedicated sub-model.\n\nThe abstract from the paper is the following:\n\n*Recent advancements in automatic speech translation have dramatically expanded language coverage, improved multimodal capabilities, and enabled a wide range of tasks and functionalities. That said, large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model\u2014SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one\u2019s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b07c43ff-6602-46da-a4a3-ab5c3251164a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b07c43ff-6602-46da-a4a3-ab5c3251164a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"af0bf3ef-b81a-4a32-8c6f-5ae79741c01d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e9197f4c-9eb9-480b-85e1-a908d08756c9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"539d35860868f918b8be0b56a5e51753bea3cff3bac26fb4538bf7585a567744\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a50d8c2a-862f-4a64-b2d0-def847a2ead5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"325a8ef556a08a12499522197b9defa003d2d2842b57cbc64fa2af0d72006f30\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1669, \"end_char_idx\": 5084, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "ref_doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "chroma:document": "First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model\u2014SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one\u2019s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work\u2014including models, code, and a watermark detector\u2014are publicly released and accessible at the link below.*\n\n## Usage\n\nIn the following example, we'll load an Arabic audio sample and an English text sample and convert them into Russian speech and French text.\n\nFirst, load the processor and a checkpoint of the model:\n\n```python\n>>> from transformers import AutoProcessor, SeamlessM4Tv2Model\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n>>> model = SeamlessM4Tv2Model.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nYou can seamlessly use this model on text or on audio, to generated either translated text or translated audio.\n\nHere is how to use the processor to process text and audio:\n\n```python\n>>> # let's load an audio sample from an Arabic speech corpus\n>>> from datasets import load_dataset\n>>> dataset = load_dataset(\"arabic_speech_corpus\", split=\"test\", streaming=True)\n>>> audio_sample = next(iter(dataset))[\"audio\"]\n\n>>> # now, process it\n>>> audio_inputs = processor(audios=audio_sample[\"array\"], return_tensors=\"pt\")\n\n>>> # now, process some English text as well\n>>> text_inputs = processor(text = \"Hello, my dog is cute\", src_lang=\"eng\", return_tensors=\"pt\")\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a50d8c2a-862f-4a64-b2d0-def847a2ead5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a50d8c2a-862f-4a64-b2d0-def847a2ead5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"af0bf3ef-b81a-4a32-8c6f-5ae79741c01d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b07c43ff-6602-46da-a4a3-ab5c3251164a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eb8f8e04ca46f92288b9e4bc0570d45b3535967ee4543485022d8b06d1b602b9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9ef5161d-1eb1-401f-af04-630eee19a5d0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4fa031276ab73fc812a1556270b865cb0bca0c8aa5377fb2dc0ae98808c78b04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3437, \"end_char_idx\": 6469, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "ref_doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "chroma:document": "Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work\u2014including models, code, and a watermark detector\u2014are publicly released and accessible at the link below.*\n\n## Usage\n\nIn the following example, we'll load an Arabic audio sample and an English text sample and convert them into Russian speech and French text.\n\nFirst, load the processor and a checkpoint of the model:\n\n```python\n>>> from transformers import AutoProcessor, SeamlessM4Tv2Model\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n>>> model = SeamlessM4Tv2Model.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nYou can seamlessly use this model on text or on audio, to generated either translated text or translated audio.\n\nHere is how to use the processor to process text and audio:\n\n```python\n>>> # let's load an audio sample from an Arabic speech corpus\n>>> from datasets import load_dataset\n>>> dataset = load_dataset(\"arabic_speech_corpus\", split=\"test\", streaming=True)\n>>> audio_sample = next(iter(dataset))[\"audio\"]\n\n>>> # now, process it\n>>> audio_inputs = processor(audios=audio_sample[\"array\"], return_tensors=\"pt\")\n\n>>> # now, process some English text as well\n>>> text_inputs = processor(text = \"Hello, my dog is cute\", src_lang=\"eng\", return_tensors=\"pt\")\n```\n\n\n### Speech\n\n[`SeamlessM4Tv2Model`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:\n\n```python\n>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n```\n\nWith basically the same code, I've translated English text and Arabic speech to Russian speech samples.\n\n### Text\n\nSimilarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4Tv2Model.generate`].\nThis time, let's translate to French.\n\n```python \n>>> # from audio\n>>> output_tokens = model.generate(**audio_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n\n>>> # from text\n>>> output_tokens = model.generate(**text_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n```\n\n### Tips\n\n\n#### 1. Use dedicated models\n\n[`SeamlessM4Tv2Model`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9ef5161d-1eb1-401f-af04-630eee19a5d0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9ef5161d-1eb1-401f-af04-630eee19a5d0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"af0bf3ef-b81a-4a32-8c6f-5ae79741c01d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a50d8c2a-862f-4a64-b2d0-def847a2ead5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ea4419864f8cf8cd666d8dd7c911f1dff2d7762bd26045c18e49ecebeccc14b3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"82a16217-0075-44bd-ad27-38316242328e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7a5fa43e44f764b39e5975ea825b2b3123485478864f4081c62a8953a4c2a6cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5087, \"end_char_idx\": 8385, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "ref_doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "chroma:document": "### Speech\n\n[`SeamlessM4Tv2Model`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:\n\n```python\n>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n```\n\nWith basically the same code, I've translated English text and Arabic speech to Russian speech samples.\n\n### Text\n\nSimilarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4Tv2Model.generate`].\nThis time, let's translate to French.\n\n```python \n>>> # from audio\n>>> output_tokens = model.generate(**audio_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n\n>>> # from text\n>>> output_tokens = model.generate(**text_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n```\n\n### Tips\n\n\n#### 1. Use dedicated models\n\n[`SeamlessM4Tv2Model`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.\nFor example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: \n\n```python\n>>> from transformers import SeamlessM4Tv2ForSpeechToSpeech\n>>> model = SeamlessM4Tv2ForSpeechToSpeech.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nOr you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.\n\n```python\n>>> from transformers import SeamlessM4Tv2ForTextToText\n>>> model = SeamlessM4Tv2ForTextToText.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nFeel free to try out [`SeamlessM4Tv2ForSpeechToText`] and [`SeamlessM4Tv2ForTextToSpeech`] as well.\n\n#### 2. Change the speaker identity\n\nYou have the possibility to change the speaker used for speech synthesis with the `speaker_id` argument. Some `speaker_id` works better than other for some languages!\n\n#### 3. Change the generation strategy\n\nYou can use different [generation strategies](../generation_strategies) for text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, text_do_sample=True)` which will perform multinomial beam-search decoding on the text model. Note that speech generation only supports greedy - by default - or multinomial sampling, which can be used with e.g. `.generate(..., speech_do_sample=True, speech_temperature=0.6)`.\n\n#### 4. Generate speech and text at the same time\n\nUse `return_intermediate_token_ids=True` with [`SeamlessM4Tv2Model`] to return both speech and text !\n\n## Model architecture\n\nSeamlessM4T-v2 features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as \"unit tokens,\" from the translated text."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 82a16217-0075-44bd-ad27-38316242328e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"82a16217-0075-44bd-ad27-38316242328e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"af0bf3ef-b81a-4a32-8c6f-5ae79741c01d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9ef5161d-1eb1-401f-af04-630eee19a5d0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b761eb8710fc563499cd5b4d68bf4798f4838e9f6e0a192189cd4814c38e2245\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9509091c-015b-4a43-8144-ef125c9927f0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1930bd56ca3593f025900111e8145bad1e17ab45ace2f0dcf739e1bddd907cab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6933, \"end_char_idx\": 10436, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "ref_doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "chroma:document": "```python\n>>> from transformers import SeamlessM4Tv2ForTextToText\n>>> model = SeamlessM4Tv2ForTextToText.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nFeel free to try out [`SeamlessM4Tv2ForSpeechToText`] and [`SeamlessM4Tv2ForTextToSpeech`] as well.\n\n#### 2. Change the speaker identity\n\nYou have the possibility to change the speaker used for speech synthesis with the `speaker_id` argument. Some `speaker_id` works better than other for some languages!\n\n#### 3. Change the generation strategy\n\nYou can use different [generation strategies](../generation_strategies) for text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, text_do_sample=True)` which will perform multinomial beam-search decoding on the text model. Note that speech generation only supports greedy - by default - or multinomial sampling, which can be used with e.g. `.generate(..., speech_do_sample=True, speech_temperature=0.6)`.\n\n#### 4. Generate speech and text at the same time\n\nUse `return_intermediate_token_ids=True` with [`SeamlessM4Tv2Model`] to return both speech and text !\n\n## Model architecture\n\nSeamlessM4T-v2 features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as \"unit tokens,\" from the translated text.\n\nEach modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.\n\n### Difference with SeamlessM4T-v1\n\nThe architecture of this new version differs from the first in a few aspects:\n\n#### Improvements on the second-pass model\n\nThe second seq2seq model, named text-to-unit model, is now non-auto regressive, meaning that it computes units in a **single forward pass**. This achievement is made possible by:\n- the use of **character-level embeddings**, meaning that each character of the predicted translated text has its own embeddings, which are then used to predict the unit tokens.\n- the use of an intermediate duration predictor, that predicts speech duration at the **character-level** on the predicted translated text.\n- the use of a new text-to-unit decoder mixing convolutions and self-attention to handle longer context.\n\n#### Difference in the speech encoder\n\nThe speech encoder, which is used during the first-pass generation process to predict the translated text, differs mainly from the previous speech encoder through these mechanisms:\n- the use of chunked attention mask to prevent attention across chunks, ensuring that each position attends only to positions within its own chunk and a fixed number of previous chunks.\n- the use of relative position embeddings which only considers distance between sequence elements rather than absolute positions. Please refer to [Self-Attentionwith Relative Position Representations (Shaw et al.)](https://arxiv.org/abs/1803.02155) for more details.\n- the use of a causal depth-wise convolution instead of a non-causal one.\n\n### Generation process\n\nHere's how the generation process works:\n\n- Input text or speech is processed through its specific encoder.\n- A decoder creates text tokens in the desired language.\n- If speech generation is required, the second seq2seq model, generates unit tokens in an non auto-regressive way."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9509091c-015b-4a43-8144-ef125c9927f0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9509091c-015b-4a43-8144-ef125c9927f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"af0bf3ef-b81a-4a32-8c6f-5ae79741c01d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"82a16217-0075-44bd-ad27-38316242328e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b0930f04b1126154ea8dd5221b6e36f87fd077fbd5917bdc161359a1c559bb44\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8623, \"end_char_idx\": 11248, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "ref_doc_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "chroma:document": "### Difference with SeamlessM4T-v1\n\nThe architecture of this new version differs from the first in a few aspects:\n\n#### Improvements on the second-pass model\n\nThe second seq2seq model, named text-to-unit model, is now non-auto regressive, meaning that it computes units in a **single forward pass**. This achievement is made possible by:\n- the use of **character-level embeddings**, meaning that each character of the predicted translated text has its own embeddings, which are then used to predict the unit tokens.\n- the use of an intermediate duration predictor, that predicts speech duration at the **character-level** on the predicted translated text.\n- the use of a new text-to-unit decoder mixing convolutions and self-attention to handle longer context.\n\n#### Difference in the speech encoder\n\nThe speech encoder, which is used during the first-pass generation process to predict the translated text, differs mainly from the previous speech encoder through these mechanisms:\n- the use of chunked attention mask to prevent attention across chunks, ensuring that each position attends only to positions within its own chunk and a fixed number of previous chunks.\n- the use of relative position embeddings which only considers distance between sequence elements rather than absolute positions. Please refer to [Self-Attentionwith Relative Position Representations (Shaw et al.)](https://arxiv.org/abs/1803.02155) for more details.\n- the use of a causal depth-wise convolution instead of a non-causal one.\n\n### Generation process\n\nHere's how the generation process works:\n\n- Input text or speech is processed through its specific encoder.\n- A decoder creates text tokens in the desired language.\n- If speech generation is required, the second seq2seq model, generates unit tokens in an non auto-regressive way.\n- These unit tokens are then passed through the final vocoder to produce the actual speech.\n\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## SeamlessM4Tv2Model\n\n[[autodoc]] SeamlessM4Tv2Model\n    - generate\n\n\n## SeamlessM4Tv2ForTextToSpeech\n\n[[autodoc]] SeamlessM4Tv2ForTextToSpeech\n    - generate\n\n\n## SeamlessM4Tv2ForSpeechToSpeech\n\n[[autodoc]] SeamlessM4Tv2ForSpeechToSpeech\n    - generate\n\n\n## SeamlessM4Tv2ForTextToText\n\n[[autodoc]] transformers.SeamlessM4Tv2ForTextToText\n    - forward\n    - generate\n\n## SeamlessM4Tv2ForSpeechToText\n\n[[autodoc]] transformers.SeamlessM4Tv2ForSpeechToText\n    - forward\n    - generate\n\n## SeamlessM4Tv2Config\n\n[[autodoc]] SeamlessM4Tv2Config"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 65623014-e540-43f0-8615-e40468d9ea0c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"65623014-e540-43f0-8615-e40468d9ea0c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"efbf1f99-5bcc-416a-a4a3-8ca60647a679\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3190299d5fce31c7d896ea5b4190690ffc9b3c8c3d4d5769212bf67361cd974b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5c674773-6177-49de-a7e1-c4357e6728ae\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aa255320c91e0db463d18ac19d0834b0ae60749832dfa14119603bb8193913da\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2941, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "doc_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "ref_doc_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "chroma:document": "# ViTMSN\n\n## Overview\n\nThe ViTMSN model was proposed in [Masked Siamese Networks for Label-Efficient Learning](https://arxiv.org/abs/2204.07141) by Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes,\nPascal Vincent, Armand Joulin, Michael Rabbat, Nicolas Ballas. The paper presents a joint-embedding architecture to match the prototypes\nof masked patches with that of the unmasked patches. With this setup, their method yields excellent performance in the low-shot and extreme low-shot\nregimes.\n\nThe abstract from the paper is the following:\n\n*We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our\napproach matches the representation of an image view containing randomly masked patches to the representation of the original\nunmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the\nunmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures,\nwhile producing representations of a high semantic level that perform competitively on low-shot image classification. For instance,\non ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy,\nand with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark.*\n\n<img src=\"https://i.ibb.co/W6PQMdC/Screenshot-2022-09-13-at-9-08-40-AM.png\" alt=\"drawing\" width=\"600\"/> \n\n<small> MSN architecture. Taken from the <a href=\"https://arxiv.org/abs/2204.07141\">original paper.</a> </small>\n\nThis model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/facebookresearch/msn). \n\n## Usage tips\n\n- MSN (masked siamese networks) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training\nobjective is to match the prototypes assigned to the unmasked views of the images to that of the masked views of the same images.\n- The authors have only released pre-trained weights of the backbone (ImageNet-1k pre-training). So, to use that on your own image classification dataset,\nuse the [`ViTMSNForImageClassification`] class which is initialized from [`ViTMSNModel`]. Follow\n[this notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb) for a detailed tutorial on fine-tuning.\n- MSN is particularly useful in the low-shot and extreme low-shot regimes. Notably, it achieves 75.7% top-1 accuracy with only 1% of ImageNet-1K\nlabels when fine-tuned.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5c674773-6177-49de-a7e1-c4357e6728ae
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5c674773-6177-49de-a7e1-c4357e6728ae\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"efbf1f99-5bcc-416a-a4a3-8ca60647a679\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3190299d5fce31c7d896ea5b4190690ffc9b3c8c3d4d5769212bf67361cd974b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"65623014-e540-43f0-8615-e40468d9ea0c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f5940b0f1fd37c3b961c35d8f0643586bc1087079573dbe966f8c9254e97d34b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e753cd48-3f44-4d64-a22d-c0cc7c3e37e5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f18e90a610b2c2bf2562ac4e6bb1b4e04c4470ecc99d5f7d7d1d92a7709b5604\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1452, \"end_char_idx\": 3904, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "doc_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "ref_doc_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "chroma:document": "*\n\n<img src=\"https://i.ibb.co/W6PQMdC/Screenshot-2022-09-13-at-9-08-40-AM.png\" alt=\"drawing\" width=\"600\"/> \n\n<small> MSN architecture. Taken from the <a href=\"https://arxiv.org/abs/2204.07141\">original paper.</a> </small>\n\nThis model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/facebookresearch/msn). \n\n## Usage tips\n\n- MSN (masked siamese networks) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training\nobjective is to match the prototypes assigned to the unmasked views of the images to that of the masked views of the same images.\n- The authors have only released pre-trained weights of the backbone (ImageNet-1k pre-training). So, to use that on your own image classification dataset,\nuse the [`ViTMSNForImageClassification`] class which is initialized from [`ViTMSNModel`]. Follow\n[this notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb) for a detailed tutorial on fine-tuning.\n- MSN is particularly useful in the low-shot and extreme low-shot regimes. Notably, it achieves 75.7% top-1 accuracy with only 1% of ImageNet-1K\nlabels when fine-tuned.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTMSNForImageClassification\nmodel = ViTMSNForImageClassification.from_pretrained(\"facebook/vit-msn-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-msn-base` model, we saw the following speedups during inference."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e753cd48-3f44-4d64-a22d-c0cc7c3e37e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e753cd48-3f44-4d64-a22d-c0cc7c3e37e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"efbf1f99-5bcc-416a-a4a3-8ca60647a679\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3190299d5fce31c7d896ea5b4190690ffc9b3c8c3d4d5769212bf67361cd974b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5c674773-6177-49de-a7e1-c4357e6728ae\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a42f440f1be22aa93224a5a436e55275441ce2799536995499343d1737158d14\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2349, \"end_char_idx\": 5668, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "doc_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "ref_doc_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "chroma:document": "Follow\n[this notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb) for a detailed tutorial on fine-tuning.\n- MSN is particularly useful in the low-shot and extreme low-shot regimes. Notably, it achieves 75.7% top-1 accuracy with only 1% of ImageNet-1K\nlabels when fine-tuned.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTMSNForImageClassification\nmodel = ViTMSNForImageClassification.from_pretrained(\"facebook/vit-msn-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-msn-base` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         7 |                                         6 |                      1.17 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViT MSN.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ViTMSNForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ViTMSNConfig\n\n[[autodoc]] ViTMSNConfig\n\n## ViTMSNModel\n\n[[autodoc]] ViTMSNModel\n    - forward\n\n## ViTMSNForImageClassification\n\n[[autodoc]] ViTMSNForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 778078ac-c723-496a-b3ec-b6d7bd950574
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"778078ac-c723-496a-b3ec-b6d7bd950574\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"91271b29-e5e4-4e37-9ae0-3c35641e33a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e25644105ec2643710c168684894bbc3a9e2bf1f03240f4133eb2e748ec8217b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5729b037-fecb-4f13-9ad7-62a0bfef1023\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2f6bedb1c220372b107306358a0681d3b21299a7c7f5a177d8900ea26200c0a6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3482, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "doc_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "ref_doc_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "chroma:document": "# OneFormer\n\n## Overview\n\nThe OneFormer model was proposed in [OneFormer: One Transformer to Rule Universal Image Segmentation](https://arxiv.org/abs/2211.06220) by Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi. OneFormer is a universal image segmentation framework that can be trained on a single panoptic dataset to perform semantic, instance, and panoptic segmentation tasks. OneFormer uses a task token to condition the model on the task in focus, making the architecture task-guided for training, and task-dynamic for inference.\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/oneformer_teaser.png\"/>\n\nThe abstract from the paper is the following:\n\n*Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible.*\n\nThe figure below illustrates the architecture of OneFormer. Taken from the [original paper](https://arxiv.org/abs/2211.06220).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/oneformer_architecture.png\"/>\n\nThis model was contributed by [Jitesh Jain](https://huggingface.co/praeclarumjj3). The original code can be found [here](https://github.com/SHI-Labs/OneFormer).\n\n## Usage tips\n\n-  OneFormer requires two inputs during inference: *image* and *task token*. \n- During training, OneFormer only uses panoptic annotations.\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  `get_num_masks` function inside in the `OneFormerLoss` class of `modeling_oneformer.py`. When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/SHI-Labs/OneFormer/blob/33ebb56ed34f970a30ae103e786c0cb64c653d9a/oneformer/modeling/criterion.py#L287)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5729b037-fecb-4f13-9ad7-62a0bfef1023
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5729b037-fecb-4f13-9ad7-62a0bfef1023\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"91271b29-e5e4-4e37-9ae0-3c35641e33a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e25644105ec2643710c168684894bbc3a9e2bf1f03240f4133eb2e748ec8217b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"778078ac-c723-496a-b3ec-b6d7bd950574\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fb99f3051e6bb3bcf462cdb3bf4b6626efaf567cf080eb68977e4836a4b8d5ed\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"75f47238-1434-40f0-a58c-70f6a62fd579\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f1b33ed8f1a7972823247100188125b82de29ee031ecf2af4efc41f862f06e5a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1815, \"end_char_idx\": 4766, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "doc_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "ref_doc_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "chroma:document": "Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible.*\n\nThe figure below illustrates the architecture of OneFormer. Taken from the [original paper](https://arxiv.org/abs/2211.06220).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/oneformer_architecture.png\"/>\n\nThis model was contributed by [Jitesh Jain](https://huggingface.co/praeclarumjj3). The original code can be found [here](https://github.com/SHI-Labs/OneFormer).\n\n## Usage tips\n\n-  OneFormer requires two inputs during inference: *image* and *task token*. \n- During training, OneFormer only uses panoptic annotations.\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  `get_num_masks` function inside in the `OneFormerLoss` class of `modeling_oneformer.py`. When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/SHI-Labs/OneFormer/blob/33ebb56ed34f970a30ae103e786c0cb64c653d9a/oneformer/modeling/criterion.py#L287).\n- One can use [`OneFormerProcessor`] to prepare input images and task inputs for the model and optional targets for the model. [`OneformerProcessor`] wraps [`OneFormerImageProcessor`] and [`CLIPTokenizer`] into a single instance to both prepare the images and encode the task inputs.\n- To get the final segmentation, depending on the task, you can call [`~OneFormerProcessor.post_process_semantic_segmentation`] or [`~OneFormerImageProcessor.post_process_instance_segmentation`] or [`~OneFormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`OneFormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with OneFormer.\n\n- Demo notebooks regarding inference + fine-tuning on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/OneFormer).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 75f47238-1434-40f0-a58c-70f6a62fd579
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"75f47238-1434-40f0-a58c-70f6a62fd579\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"91271b29-e5e4-4e37-9ae0-3c35641e33a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e25644105ec2643710c168684894bbc3a9e2bf1f03240f4133eb2e748ec8217b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5729b037-fecb-4f13-9ad7-62a0bfef1023\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"939a1c7e988168925c49b6d2d1d15560056dc37d89e4db4a9aef4c76cf16a72d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3197, \"end_char_idx\": 5445, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "doc_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "ref_doc_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "chroma:document": "When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/SHI-Labs/OneFormer/blob/33ebb56ed34f970a30ae103e786c0cb64c653d9a/oneformer/modeling/criterion.py#L287).\n- One can use [`OneFormerProcessor`] to prepare input images and task inputs for the model and optional targets for the model. [`OneformerProcessor`] wraps [`OneFormerImageProcessor`] and [`CLIPTokenizer`] into a single instance to both prepare the images and encode the task inputs.\n- To get the final segmentation, depending on the task, you can call [`~OneFormerProcessor.post_process_semantic_segmentation`] or [`~OneFormerImageProcessor.post_process_instance_segmentation`] or [`~OneFormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`OneFormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with OneFormer.\n\n- Demo notebooks regarding inference + fine-tuning on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/OneFormer).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## OneFormer specific outputs\n\n[[autodoc]] models.oneformer.modeling_oneformer.OneFormerModelOutput\n\n[[autodoc]] models.oneformer.modeling_oneformer.OneFormerForUniversalSegmentationOutput\n\n## OneFormerConfig\n\n[[autodoc]] OneFormerConfig\n\n## OneFormerImageProcessor\n\n[[autodoc]] OneFormerImageProcessor\n    - preprocess\n    - encode_inputs\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## OneFormerProcessor\n\n[[autodoc]] OneFormerProcessor\n\n## OneFormerModel\n\n[[autodoc]] OneFormerModel\n    - forward\n\n## OneFormerForUniversalSegmentation\n\n[[autodoc]] OneFormerForUniversalSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b87e35e2-5d9d-48de-97ea-9767a64c92a2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sew", "title": "SEW", "tokens": 629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b87e35e2-5d9d-48de-97ea-9767a64c92a2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sew\", \"title\": \"SEW\", \"tokens\": 629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a1ec8e58-c5b8-4a82-ba89-c2b775adb4eb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sew\", \"title\": \"SEW\", \"tokens\": 629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"48c93e7aacb4789c8c0d458461eafea0471b5599c273e6d4b3cedc401536e322\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1811, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a1ec8e58-c5b8-4a82-ba89-c2b775adb4eb", "doc_id": "a1ec8e58-c5b8-4a82-ba89-c2b775adb4eb", "ref_doc_id": "a1ec8e58-c5b8-4a82-ba89-c2b775adb4eb", "chroma:document": "# SEW\n\n## Overview\n\nSEW (Squeezed and Efficient Wav2Vec) was proposed in [Performance-Efficiency Trade-offs in Unsupervised Pre-training\nfor Speech Recognition](https://arxiv.org/abs/2109.06870) by Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q.\nWeinberger, Yoav Artzi.\n\nThe abstract from the paper is the following:\n\n*This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition\n(ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance\nand its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a\npre-trained model architecture with significant improvements along both performance and efficiency dimensions across a\nvariety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x\ninference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference\ntime, SEW reduces word error rate by 25-50% across different model sizes.*\n\nThis model was contributed by [anton-l](https://huggingface.co/anton-l).\n\n## Usage tips\n\n- SEW is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- SEWForCTC is fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded using\n  [`Wav2Vec2CTCTokenizer`].\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## SEWConfig\n\n[[autodoc]] SEWConfig\n\n## SEWModel\n\n[[autodoc]] SEWModel\n    - forward\n\n## SEWForCTC\n\n[[autodoc]] SEWForCTC\n    - forward\n\n## SEWForSequenceClassification\n\n[[autodoc]] SEWForSequenceClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 99181b5d-8740-4f9a-a448-43e8f47227bf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/altclip", "title": "AltCLIP", "tokens": 1185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"99181b5d-8740-4f9a-a448-43e8f47227bf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69c7558e-5c4c-4a47-bad9-61284865bdaa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f5f33c1273d0ef1283fb7b8cb8974be2aa8c1405422beaed3e59413eb0eafd8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9d1cde9b-8436-4bf5-bba0-0be9f862f205\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"592ccbb7cfd3ec4ff63797fb6a80e4e7dd13e4036ff74e872b8fd85bb8c09661\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2996, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69c7558e-5c4c-4a47-bad9-61284865bdaa", "doc_id": "69c7558e-5c4c-4a47-bad9-61284865bdaa", "ref_doc_id": "69c7558e-5c4c-4a47-bad9-61284865bdaa", "chroma:document": "# AltCLIP\n\n## Overview\n\nThe AltCLIP model was proposed in [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://arxiv.org/abs/2211.06679v2) by Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, Ledell Wu. AltCLIP\n(Altering the Language Encoder in CLIP) is a neural network trained on a variety of image-text and text-text pairs. By switching CLIP's\ntext encoder with a pretrained multilingual text encoder XLM-R, we could obtain very close performances with CLIP on almost all tasks, and extended original CLIP's capabilities such as multilingual understanding.\n\nThe abstract from the paper is the following:\n\n*In this work, we present a conceptually simple and effective method to train a strong bilingual multimodal representation model. \nStarting from the pretrained multimodal representation model CLIP released by OpenAI, we switched its text encoder with a pretrained \nmultilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of \nteacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art \nperformances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with \nCLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding.*\n\nThis model was contributed by [jongjyh](https://huggingface.co/jongjyh).\n\n## Usage tips and example\n\nThe usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention\nand we take the [CLS] token in XLM-R to represent text embedding.\n\nAltCLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image\nclassification. AltCLIP uses a ViT like transformer to get visual features and a bidirectional language model to get the text\nfeatures. Both the text and visual features are then projected to a latent space with identical dimension. The dot\nproduct between the projected image and text features is then used as a similar score.\n\nTo feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,\nwhich are then linearly embedded. A [CLS] token is added to serve as representation of an entire image. The authors\nalso add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder.\nThe [`CLIPImageProcessor`] can be used to resize (or rescale) and normalize images for the model.\n\nThe [`AltCLIPProcessor`] wraps a [`CLIPImageProcessor`] and a [`XLMRobertaTokenizer`] into a single instance to both\nencode the text and prepare the images. The following example shows how to get the image-text similarity scores using\n[`AltCLIPProcessor`] and [`AltCLIPModel`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9d1cde9b-8436-4bf5-bba0-0be9f862f205
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/altclip", "title": "AltCLIP", "tokens": 1185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9d1cde9b-8436-4bf5-bba0-0be9f862f205\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69c7558e-5c4c-4a47-bad9-61284865bdaa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f5f33c1273d0ef1283fb7b8cb8974be2aa8c1405422beaed3e59413eb0eafd8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"99181b5d-8740-4f9a-a448-43e8f47227bf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b1ea86842c272b8e69db36ecbb603c80d80ba84983bf0300d3c4aaa1d0bbce47\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1286, \"end_char_idx\": 4294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69c7558e-5c4c-4a47-bad9-61284865bdaa", "doc_id": "69c7558e-5c4c-4a47-bad9-61284865bdaa", "ref_doc_id": "69c7558e-5c4c-4a47-bad9-61284865bdaa", "chroma:document": "Further, we obtain very close performances with \nCLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding.*\n\nThis model was contributed by [jongjyh](https://huggingface.co/jongjyh).\n\n## Usage tips and example\n\nThe usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention\nand we take the [CLS] token in XLM-R to represent text embedding.\n\nAltCLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image\nclassification. AltCLIP uses a ViT like transformer to get visual features and a bidirectional language model to get the text\nfeatures. Both the text and visual features are then projected to a latent space with identical dimension. The dot\nproduct between the projected image and text features is then used as a similar score.\n\nTo feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,\nwhich are then linearly embedded. A [CLS] token is added to serve as representation of an entire image. The authors\nalso add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder.\nThe [`CLIPImageProcessor`] can be used to resize (or rescale) and normalize images for the model.\n\nThe [`AltCLIPProcessor`] wraps a [`CLIPImageProcessor`] and a [`XLMRobertaTokenizer`] into a single instance to both\nencode the text and prepare the images. The following example shows how to get the image-text similarity scores using\n[`AltCLIPProcessor`] and [`AltCLIPModel`].\n\n```python\n>>> from PIL import Image\n>>> import requests\n\n>>> from transformers import AltCLIPModel, AltCLIPProcessor\n\n>>> model = AltCLIPModel.from_pretrained(\"BAAI/AltCLIP\")\n>>> processor = AltCLIPProcessor.from_pretrained(\"BAAI/AltCLIP\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> inputs = processor(text=[\"a photo of a cat\", \"a photo of a dog\"], images=image, return_tensors=\"pt\", padding=True)\n\n>>> outputs = model(**inputs)\n>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score\n>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities\n```\n\n<Tip>\n\nThis model is based on `CLIPModel`, use it like you would use the original [CLIP](clip).\n\n</Tip>\n\n## AltCLIPConfig\n\n[[autodoc]] AltCLIPConfig\n    - from_text_vision_configs\n\n## AltCLIPTextConfig\n\n[[autodoc]] AltCLIPTextConfig\n\n## AltCLIPVisionConfig\n\n[[autodoc]] AltCLIPVisionConfig\n\n## AltCLIPProcessor\n\n[[autodoc]] AltCLIPProcessor\n\n## AltCLIPModel\n\n[[autodoc]] AltCLIPModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## AltCLIPTextModel\n\n[[autodoc]] AltCLIPTextModel\n    - forward\n\n## AltCLIPVisionModel\n\n[[autodoc]] AltCLIPVisionModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1b1c04d0-7ba3-43b8-b265-f5fc1ace166f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/paligemma", "title": "PaliGemma", "tokens": 1039, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1b1c04d0-7ba3-43b8-b265-f5fc1ace166f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f780dda-f343-41c4-a076-136ed706b2ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bedc083f2570c825df7e8f8b4f61b04ab2a997667c2ed1cbe90abd7b6c28d376\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"88bb438e-d994-4dc4-a234-2171507802b6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c0a9a223d9d212987dc2f2a45726c2147d1b52cf1f64272755adcf7d43a0848\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3094, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f780dda-f343-41c4-a076-136ed706b2ad", "doc_id": "3f780dda-f343-41c4-a076-136ed706b2ad", "ref_doc_id": "3f780dda-f343-41c4-a076-136ed706b2ad", "chroma:document": "# PaliGemma\n\n## Overview\n\nThe PaliGemma model was proposed in [PaliGemma \u2013 Google's Cutting-Edge Open Vision Language Model](https://huggingface.co/blog/paligemma) by Google. It is a 3B vision-language model composed by a [SigLIP](siglip) vision encoder and a [Gemma](gemma) language decoder linked by a multimodal linear projection. It cuts an image into a fixed number of VIT tokens and prepends it to an optional prompt. One particularity is that the model uses full block attention on all the image tokens plus the input text tokens. It comes in 3 resolutions, 224x224, 448x448 and 896x896 with 3 base models, with 55 fine-tuned versions for different tasks, and 2 mix models.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/paligemma/paligemma_arch.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> PaliGemma architecture. Taken from the <a href=\"https://huggingface.co/blog/paligemma\">blog post.</a> </small>\n\nThis model was contributed by [Molbap](https://huggingface.co/Molbap).\n\n## Usage tips\n\nInference with PaliGemma can be performed as follows:\n\n```python\nfrom transformers import AutoProcessor, PaliGemmaForConditionalGeneration\n\nmodel_id = \"google/paligemma-3b-mix-224\"\nmodel = PaliGemmaForConditionalGeneration.from_pretrained(model_id)\nprocessor = AutoProcessor.from_pretrained(model_id)\n\nprompt = \"What is on the flower?\"\nimage_file = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=true\"\nraw_image = Image.open(requests.get(image_file, stream=True).raw)\ninputs = processor(prompt, raw_image, return_tensors=\"pt\")\noutput = model.generate(**inputs, max_new_tokens=20)\n\nprint(processor.decode(output[0], skip_special_tokens=True)[len(prompt):])\n```\n\n- PaliGemma is not meant for conversational use, and it works best when fine-tuning to a specific use case. Some downstream tasks on which PaliGemma can be fine-tuned include image captioning, visual question answering (VQA), object detection, referring expression segmentation and document understanding.\n- One can use `PaliGemmaProcessor` to prepare images, text and optional labels for the model. When fine-tuning a PaliGemma model, the `suffix` argument can be passed to the processor which creates the `labels` for the model:\n\n```python\nprompt = \"What is on the flower?\"\nanswer = \"a bee\"\ninputs = processor(text=prompt, images=raw_image, suffix=answer, return_tensors=\"pt\")\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with PaliGemma. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A blog post introducing all the features of PaliGemma can be found [here](https://huggingface.co/blog/paligemma).\n- Demo notebooks on how to fine-tune PaliGemma for VQA with the Trainer API along with inference can be found [here](https://github.com/huggingface/notebooks/tree/main/examples/paligemma)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 88bb438e-d994-4dc4-a234-2171507802b6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/paligemma", "title": "PaliGemma", "tokens": 1039, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"88bb438e-d994-4dc4-a234-2171507802b6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f780dda-f343-41c4-a076-136ed706b2ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bedc083f2570c825df7e8f8b4f61b04ab2a997667c2ed1cbe90abd7b6c28d376\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1b1c04d0-7ba3-43b8-b265-f5fc1ace166f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"29b13b939a85a7a340c15623c2a63500b3448408487d054540e59efd59a851fb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1859, \"end_char_idx\": 3508, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f780dda-f343-41c4-a076-136ed706b2ad", "doc_id": "3f780dda-f343-41c4-a076-136ed706b2ad", "ref_doc_id": "3f780dda-f343-41c4-a076-136ed706b2ad", "chroma:document": "Some downstream tasks on which PaliGemma can be fine-tuned include image captioning, visual question answering (VQA), object detection, referring expression segmentation and document understanding.\n- One can use `PaliGemmaProcessor` to prepare images, text and optional labels for the model. When fine-tuning a PaliGemma model, the `suffix` argument can be passed to the processor which creates the `labels` for the model:\n\n```python\nprompt = \"What is on the flower?\"\nanswer = \"a bee\"\ninputs = processor(text=prompt, images=raw_image, suffix=answer, return_tensors=\"pt\")\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with PaliGemma. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A blog post introducing all the features of PaliGemma can be found [here](https://huggingface.co/blog/paligemma).\n- Demo notebooks on how to fine-tune PaliGemma for VQA with the Trainer API along with inference can be found [here](https://github.com/huggingface/notebooks/tree/main/examples/paligemma).\n- Demo notebooks on how to fine-tune PaliGemma on a custom dataset (receipt image -> JSON) along with inference can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/PaliGemma). \ud83c\udf0e\n\n## PaliGemmaConfig\n\n[[autodoc]] PaliGemmaConfig\n\n## PaliGemmaProcessor\n\n[[autodoc]] PaliGemmaProcessor\n\n## PaliGemmaForConditionalGeneration\n\n[[autodoc]] PaliGemmaForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0d69273f-bb56-4013-b6df-70cad2a920cb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0d69273f-bb56-4013-b6df-70cad2a920cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f26ea05d-07b4-4ca8-824c-30806ce02fbe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b95a7e96913cb57cce5d38f405fab9a95088afff62ccc00a0ec909cc4d1f2278\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3392, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "ref_doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "chroma:document": "# Encoder Decoder Models\n\n## Overview\n\nThe [`EncoderDecoderModel`] can be used to initialize a sequence-to-sequence model with any\npretrained autoencoding model as the encoder and any pretrained autoregressive model as the decoder.\n\nThe effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation tasks\nwas shown in [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://arxiv.org/abs/1907.12461) by\nSascha Rothe, Shashi Narayan, Aliaksei Severyn.\n\nAfter such an [`EncoderDecoderModel`] has been trained/fine-tuned, it can be saved/loaded just like\nany other models (see the examples for more information).\n\nAn application of this architecture could be to leverage two pretrained [`BertModel`] as the encoder\nand decoder for a summarization model as was shown in: [Text Summarization with Pretrained Encoders](https://arxiv.org/abs/1908.08345) by Yang Liu and Mirella Lapata.\n\n## Randomly initializing `EncoderDecoderModel` from model configurations.\n\n[`EncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default [`BertModel`] configuration for the encoder and the default [`BertForCausalLM`] configuration for the decoder.\n\n```python\n>>> from transformers import BertConfig, EncoderDecoderConfig, EncoderDecoderModel\n\n>>> config_encoder = BertConfig()\n>>> config_decoder = BertConfig()\n\n>>> config = EncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)\n>>> model = EncoderDecoderModel(config=config)\n```\n\n## Initialising `EncoderDecoderModel` from a pretrained encoder and a pretrained decoder.\n\n[`EncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained auto-encoding model, *e.g.* BERT, can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.\nDepending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.\nInitializing [`EncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).\nTo do so, the `EncoderDecoderModel` class provides a [`EncoderDecoderModel.from_encoder_decoder_pretrained`] method.\n\n```python\n>>> from transformers import EncoderDecoderModel, BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(\"google-bert/bert-base-uncased\", \"google-bert/bert-base-uncased\")\n```\n\n## Loading an existing `EncoderDecoderModel` checkpoint and perform inference.\n\nTo load fine-tuned checkpoints of the `EncoderDecoderModel` class, [`EncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.\n\nTo perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f26ea05d-07b4-4ca8-824c-30806ce02fbe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f26ea05d-07b4-4ca8-824c-30806ce02fbe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0d69273f-bb56-4013-b6df-70cad2a920cb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e88bb2fdb3af0bb37f9240ce62105bb947bfe5a76ba187dedfec987f2496b910\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1e15aa7a-5fdf-4d24-8d77-f27e513e174d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b852675c1986f54dcd15c5d45b997ca5592027b501a1171722a88a4ea9ceabea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1692, \"end_char_idx\": 4961, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "ref_doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "chroma:document": "[`EncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained auto-encoding model, *e.g.* BERT, can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.\nDepending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.\nInitializing [`EncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).\nTo do so, the `EncoderDecoderModel` class provides a [`EncoderDecoderModel.from_encoder_decoder_pretrained`] method.\n\n```python\n>>> from transformers import EncoderDecoderModel, BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(\"google-bert/bert-base-uncased\", \"google-bert/bert-base-uncased\")\n```\n\n## Loading an existing `EncoderDecoderModel` checkpoint and perform inference.\n\nTo load fine-tuned checkpoints of the `EncoderDecoderModel` class, [`EncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.\n\nTo perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.\n\n```python\n>>> from transformers import AutoTokenizer, EncoderDecoderModel\n\n>>> # load a fine-tuned seq2seq model and corresponding tokenizer\n>>> model = EncoderDecoderModel.from_pretrained(\"patrickvonplaten/bert2bert_cnn_daily_mail\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"patrickvonplaten/bert2bert_cnn_daily_mail\")\n\n>>> # let's perform inference on a long piece of text\n>>> ARTICLE_TO_SUMMARIZE = (\n...     \"PG&E stated it scheduled the blackouts in response to forecasts for high winds \"\n...     \"amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were \"\n...     \"scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.\"\n... )\n>>> input_ids = tokenizer(ARTICLE_TO_SUMMARIZE, return_tensors=\"pt\").input_ids\n\n>>> # autoregressively generate summary (uses greedy decoding by default)\n>>> generated_ids = model.generate(input_ids)\n>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> print(generated_text)\nnearly 800 thousand customers were affected by the shutoffs. the aim is to reduce the risk of wildfires. nearly 800, 000 customers were expected to be affected by high winds amid dry conditions. pg & e said it scheduled the blackouts to last through at least midday tomorrow.\n```\n\n## Loading a PyTorch checkpoint into `TFEncoderDecoderModel`.\n\n[`TFEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a\npytorch checkpoint. Passing `from_pt=True` to this method will throw an exception."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1e15aa7a-5fdf-4d24-8d77-f27e513e174d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1e15aa7a-5fdf-4d24-8d77-f27e513e174d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f26ea05d-07b4-4ca8-824c-30806ce02fbe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6e1fc1bdfb6bf9a1d70d26b1b45639cb40fe00216303fe7ea5b7b3d641afb613\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a92776fb-4829-411e-bc83-aae3eb9c7827\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e5c66299667eee86a06fd473c110aeb101ff6e5a7e8525ef41ce2ec6f8d6674d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3187, \"end_char_idx\": 6570, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "ref_doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "chroma:document": "To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.\n\n```python\n>>> from transformers import AutoTokenizer, EncoderDecoderModel\n\n>>> # load a fine-tuned seq2seq model and corresponding tokenizer\n>>> model = EncoderDecoderModel.from_pretrained(\"patrickvonplaten/bert2bert_cnn_daily_mail\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"patrickvonplaten/bert2bert_cnn_daily_mail\")\n\n>>> # let's perform inference on a long piece of text\n>>> ARTICLE_TO_SUMMARIZE = (\n...     \"PG&E stated it scheduled the blackouts in response to forecasts for high winds \"\n...     \"amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were \"\n...     \"scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.\"\n... )\n>>> input_ids = tokenizer(ARTICLE_TO_SUMMARIZE, return_tensors=\"pt\").input_ids\n\n>>> # autoregressively generate summary (uses greedy decoding by default)\n>>> generated_ids = model.generate(input_ids)\n>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> print(generated_text)\nnearly 800 thousand customers were affected by the shutoffs. the aim is to reduce the risk of wildfires. nearly 800, 000 customers were expected to be affected by high winds amid dry conditions. pg & e said it scheduled the blackouts to last through at least midday tomorrow.\n```\n\n## Loading a PyTorch checkpoint into `TFEncoderDecoderModel`.\n\n[`TFEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a\npytorch checkpoint. Passing `from_pt=True` to this method will throw an exception. If there are only pytorch\ncheckpoints for a particular encoder-decoder model, a workaround is:\n\n```python\n>>> # a workaround to load from pytorch checkpoint\n>>> from transformers import EncoderDecoderModel, TFEncoderDecoderModel\n\n>>> _model = EncoderDecoderModel.from_pretrained(\"patrickvonplaten/bert2bert-cnn_dailymail-fp16\")\n\n>>> _model.encoder.save_pretrained(\"./encoder\")\n>>> _model.decoder.save_pretrained(\"./decoder\")\n\n>>> model = TFEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"./encoder\", \"./decoder\", encoder_from_pt=True, decoder_from_pt=True\n... )\n>>> # This is only for copying some specific attributes of this particular model.\n>>> model.config = _model.config\n```\n\n## Training\n\nOnce the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model.\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the\n`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded\ntarget sequence).\n\n```python\n>>> from transformers import BertTokenizer, EncoderDecoderModel\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(\"google-bert/bert-base-uncased\", \"google-bert/bert-base-uncased\")\n\n>>> model.config.decoder_start_token_id = tokenizer.cls_token_id\n>>> model.config.pad_token_id = tokenizer.pad_token_id\n\n>>> input_ids = tokenizer(\n...     \"The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a92776fb-4829-411e-bc83-aae3eb9c7827
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a92776fb-4829-411e-bc83-aae3eb9c7827\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1e15aa7a-5fdf-4d24-8d77-f27e513e174d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2cdaca6044c3864aa4509ca7a978528f02a7dcfebff13f4cabb3a971f28a9f97\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6bfadcc0-9529-4e83-805f-9f3b32958858\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c0f072ed16a5400a026af7494c78925cda08ed3c4904b3455cfc26033194c93d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4899, \"end_char_idx\": 8077, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "ref_doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "chroma:document": "Passing `from_pt=True` to this method will throw an exception. If there are only pytorch\ncheckpoints for a particular encoder-decoder model, a workaround is:\n\n```python\n>>> # a workaround to load from pytorch checkpoint\n>>> from transformers import EncoderDecoderModel, TFEncoderDecoderModel\n\n>>> _model = EncoderDecoderModel.from_pretrained(\"patrickvonplaten/bert2bert-cnn_dailymail-fp16\")\n\n>>> _model.encoder.save_pretrained(\"./encoder\")\n>>> _model.decoder.save_pretrained(\"./decoder\")\n\n>>> model = TFEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"./encoder\", \"./decoder\", encoder_from_pt=True, decoder_from_pt=True\n... )\n>>> # This is only for copying some specific attributes of this particular model.\n>>> model.config = _model.config\n```\n\n## Training\n\nOnce the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model.\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the\n`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded\ntarget sequence).\n\n```python\n>>> from transformers import BertTokenizer, EncoderDecoderModel\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(\"google-bert/bert-base-uncased\", \"google-bert/bert-base-uncased\")\n\n>>> model.config.decoder_start_token_id = tokenizer.cls_token_id\n>>> model.config.pad_token_id = tokenizer.pad_token_id\n\n>>> input_ids = tokenizer(\n...     \"The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side.During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was  finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft).Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.\",\n...     return_tensors=\"pt\",\n... ).input_ids\n\n>>> labels = tokenizer(\n...     \"the eiffel tower surpassed the washington monument to become the tallest structure in the world. it was the first structure to reach a height of 300 metres in paris in 1930. it is now taller than the chrysler building by 5. 2 metres ( 17 ft ) and is the second tallest free - standing structure in paris.\",\n...     return_tensors=\"pt\",\n... ).input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n```\n\nDetailed [colab](https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44_VAyKE?usp=sharing#scrollTo=ZwQIEhKOrJpl) for training.\n\nThis model was contributed by [thomwolf](https://github.com/thomwolf). This model's TensorFlow and Flax versions\nwere contributed by [ydshieh](https://github.com/ydshieh)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6bfadcc0-9529-4e83-805f-9f3b32958858
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6bfadcc0-9529-4e83-805f-9f3b32958858\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a92776fb-4829-411e-bc83-aae3eb9c7827\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b9a801f70aefeed36f908428af1ea393cf905cd2115111554ae05922f37a5509\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6571, \"end_char_idx\": 8552, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "ref_doc_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "chroma:document": "Its base is square, measuring 125 metres (410 ft) on each side.During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was  finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft).Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.\",\n...     return_tensors=\"pt\",\n... ).input_ids\n\n>>> labels = tokenizer(\n...     \"the eiffel tower surpassed the washington monument to become the tallest structure in the world. it was the first structure to reach a height of 300 metres in paris in 1930. it is now taller than the chrysler building by 5. 2 metres ( 17 ft ) and is the second tallest free - standing structure in paris.\",\n...     return_tensors=\"pt\",\n... ).input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n```\n\nDetailed [colab](https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44_VAyKE?usp=sharing#scrollTo=ZwQIEhKOrJpl) for training.\n\nThis model was contributed by [thomwolf](https://github.com/thomwolf). This model's TensorFlow and Flax versions\nwere contributed by [ydshieh](https://github.com/ydshieh).\n\n\n## EncoderDecoderConfig\n\n[[autodoc]] EncoderDecoderConfig\n\n<frameworkcontent>\n<pt>\n\n## EncoderDecoderModel\n\n[[autodoc]] EncoderDecoderModel\n    - forward\n    - from_encoder_decoder_pretrained\n\n</pt>\n<tf>\n\n## TFEncoderDecoderModel\n\n[[autodoc]] TFEncoderDecoderModel\n    - call\n    - from_encoder_decoder_pretrained\n\n</tf>\n<jax>\n\n## FlaxEncoderDecoderModel\n\n[[autodoc]] FlaxEncoderDecoderModel\n    - __call__\n    - from_encoder_decoder_pretrained\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6938536c-6a02-47d0-a228-5a8ec2947b0a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6938536c-6a02-47d0-a228-5a8ec2947b0a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"deacb9c7-75ab-464d-adff-6f38f7931701\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ef21361-6443-413f-a86e-5d9bda19ad5d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"25d2a0215c0df4b5d8ed28cb3447787b102f66870704c54698f3aba2359e5b68\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2731, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "ref_doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "chroma:document": "# SigLIP\n\n## Overview\n\nThe SigLIP model was proposed in [Sigmoid Loss for Language Image Pre-Training](https://arxiv.org/abs/2303.15343) by Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer. SigLIP proposes to replace the loss function used in [CLIP](clip) by a simple pairwise sigmoid loss. This results in better performance in terms of zero-shot classification accuracy on ImageNet.\n\nThe abstract from the paper is the following:\n\n*We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient.*\n\n## Usage tips\n\n- Usage of SigLIP is similar to [CLIP](clip). The main difference is the training loss, which does not require a global view of all the pairwise similarities of images and texts within a batch. One needs to apply the sigmoid activation function to the logits, rather than the softmax.\n- Training is supported but does not use `torch.distributed` utilities which may limit the scalability of batch size. However, DDP and FDSP works on single-node multi-gpu setup.\n- When using the standalone [`SiglipTokenizer`] or [`SiglipProcessor`], make sure to pass `padding=\"max_length\"` as that's how the model was trained.\n- To get the same results as the pipeline, a prompt template of \"This is a photo of {label}.\" should be used.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/siglip_table.jpeg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> SigLIP evaluation results compared to CLIP. Taken from the <a href=\"https://arxiv.org/abs/2303.15343\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/google-research/big_vision/tree/main).\n\n## Usage example\n\nThere are 2 main ways to use SigLIP: either using the pipeline API, which abstracts away all the complexity for you, or by using the `SiglipModel` class yourself."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1ef21361-6443-413f-a86e-5d9bda19ad5d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1ef21361-6443-413f-a86e-5d9bda19ad5d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"deacb9c7-75ab-464d-adff-6f38f7931701\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6938536c-6a02-47d0-a228-5a8ec2947b0a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b4ea411b6c4134b543ce30c25ad92ce9c44dc20963413fba7159a3b8c03e5513\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"46c8aec0-1503-4f90-8b7c-f4e993986f16\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c1dda6edc01caeedb43185c9921a9034699c21f9fc8a60bdcf49a7dbb5bd1311\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1151, \"end_char_idx\": 4239, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "ref_doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "chroma:document": "Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient.*\n\n## Usage tips\n\n- Usage of SigLIP is similar to [CLIP](clip). The main difference is the training loss, which does not require a global view of all the pairwise similarities of images and texts within a batch. One needs to apply the sigmoid activation function to the logits, rather than the softmax.\n- Training is supported but does not use `torch.distributed` utilities which may limit the scalability of batch size. However, DDP and FDSP works on single-node multi-gpu setup.\n- When using the standalone [`SiglipTokenizer`] or [`SiglipProcessor`], make sure to pass `padding=\"max_length\"` as that's how the model was trained.\n- To get the same results as the pipeline, a prompt template of \"This is a photo of {label}.\" should be used.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/siglip_table.jpeg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> SigLIP evaluation results compared to CLIP. Taken from the <a href=\"https://arxiv.org/abs/2303.15343\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/google-research/big_vision/tree/main).\n\n## Usage example\n\nThere are 2 main ways to use SigLIP: either using the pipeline API, which abstracts away all the complexity for you, or by using the `SiglipModel` class yourself.\n\n### Pipeline API\n\nThe pipeline allows to use the model in a few lines of code:\n\n```python\n>>> from transformers import pipeline\n>>> from PIL import Image\n>>> import requests\n\n>>> # load pipe\n>>> image_classifier = pipeline(task=\"zero-shot-image-classification\", model=\"google/siglip-base-patch16-224\")\n\n>>> # load image\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> # inference\n>>> candidate_labels = [\"2 cats\", \"a plane\", \"a remote\"]\n>>> outputs = image_classifier(image, candidate_labels=candidate_labels)\n>>> outputs = [{\"score\": round(output[\"score\"], 4), \"label\": output[\"label\"] } for output in outputs]\n>>> print(outputs)\n[{'score': 0.1979, 'label': '2 cats'}, {'score': 0.0, 'label': 'a remote'}, {'score': 0.0, 'label': 'a plane'}]\n```\n\n### Using the model yourself\n\nIf you want to do the pre- and postprocessing yourself, here's how to do that:\n\n```python\n>>> from PIL import Image\n>>> import requests\n>>> from transformers import AutoProcessor, AutoModel\n>>> import torch\n\n>>> model = AutoModel.from_pretrained(\"google/siglip-base-patch16-224\")\n>>> processor = AutoProcessor.from_pretrained(\"google/siglip-base-patch16-224\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> candidate_labels = [\"2 cats\", \"2 dogs\"]\n# follows the pipeline prompt template to get same results\n>>> candidate_labels = [f'This is a photo of {label}.'"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 46c8aec0-1503-4f90-8b7c-f4e993986f16
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"46c8aec0-1503-4f90-8b7c-f4e993986f16\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"deacb9c7-75ab-464d-adff-6f38f7931701\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1ef21361-6443-413f-a86e-5d9bda19ad5d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a479b257c24371ddf1d442cedaddf6ee6bed2acf1cf9a3a86e11992be5c6d4a5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4306c05c-a5bb-4668-8d4c-af45d9335822\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0ff1f7a0d2c2b1df80b7422161ad67e3a91f0a1ed985764b0a01aed5b780737\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2733, \"end_char_idx\": 5305, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "ref_doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "chroma:document": "### Pipeline API\n\nThe pipeline allows to use the model in a few lines of code:\n\n```python\n>>> from transformers import pipeline\n>>> from PIL import Image\n>>> import requests\n\n>>> # load pipe\n>>> image_classifier = pipeline(task=\"zero-shot-image-classification\", model=\"google/siglip-base-patch16-224\")\n\n>>> # load image\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> # inference\n>>> candidate_labels = [\"2 cats\", \"a plane\", \"a remote\"]\n>>> outputs = image_classifier(image, candidate_labels=candidate_labels)\n>>> outputs = [{\"score\": round(output[\"score\"], 4), \"label\": output[\"label\"] } for output in outputs]\n>>> print(outputs)\n[{'score': 0.1979, 'label': '2 cats'}, {'score': 0.0, 'label': 'a remote'}, {'score': 0.0, 'label': 'a plane'}]\n```\n\n### Using the model yourself\n\nIf you want to do the pre- and postprocessing yourself, here's how to do that:\n\n```python\n>>> from PIL import Image\n>>> import requests\n>>> from transformers import AutoProcessor, AutoModel\n>>> import torch\n\n>>> model = AutoModel.from_pretrained(\"google/siglip-base-patch16-224\")\n>>> processor = AutoProcessor.from_pretrained(\"google/siglip-base-patch16-224\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> candidate_labels = [\"2 cats\", \"2 dogs\"]\n# follows the pipeline prompt template to get same results\n>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels]\n>>> # important: we pass `padding=max_length` since the model was trained with this\n>>> inputs = processor(text=texts, images=image, padding=\"max_length\", return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> logits_per_image = outputs.logits_per_image\n>>> probs = torch.sigmoid(logits_per_image) # these are the probabilities\n>>> print(f\"{probs[0][0]:.1%} that image 0 is '{texts[0]}'\")\n31.9% that image 0 is 'a photo of 2 cats'\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SigLIP.\n\n- [Zero-shot image classification task guide](../tasks/zero_shot_image_classification_md)\n- Demo notebooks for SigLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SigLIP). \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4306c05c-a5bb-4668-8d4c-af45d9335822
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4306c05c-a5bb-4668-8d4c-af45d9335822\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"deacb9c7-75ab-464d-adff-6f38f7931701\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"46c8aec0-1503-4f90-8b7c-f4e993986f16\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0bb46e7113b19679fbda4e938414fb37e1f2deb469efd0e708e277cc962f6e80\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"927ebaf6-4d71-47a0-a1fb-058f4d876f3d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3edbd5c5dd12dd084626cea7668416fc55fb582760d2106086ede213cd3aede1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4240, \"end_char_idx\": 7141, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "ref_doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "chroma:document": "for label in candidate_labels]\n>>> # important: we pass `padding=max_length` since the model was trained with this\n>>> inputs = processor(text=texts, images=image, padding=\"max_length\", return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> logits_per_image = outputs.logits_per_image\n>>> probs = torch.sigmoid(logits_per_image) # these are the probabilities\n>>> print(f\"{probs[0][0]:.1%} that image 0 is '{texts[0]}'\")\n31.9% that image 0 is 'a photo of 2 cats'\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SigLIP.\n\n- [Zero-shot image classification task guide](../tasks/zero_shot_image_classification_md)\n- Demo notebooks for SigLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SigLIP). \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n\n## Combining SigLIP and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> import requests\n>>> from PIL import Image\n>>> from transformers import SiglipProcessor, SiglipModel\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = SiglipModel.from_pretrained(\n...     \"google/siglip-so400m-patch14-384\",\n...     attn_implementation=\"flash_attention_2\",\n...     torch_dtype=torch.float16,\n...     device_map=device,\n... )\n>>> processor = SiglipProcessor.from_pretrained(\"google/siglip-so400m-patch14-384\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> candidate_labels = [\"2 cats\", \"2 dogs\"]\n# follows the pipeline prompt template to get same results\n>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels]\n# important: we pass `padding=max_length` since the model was trained with this\n>>> inputs = processor(text=candidate_labels, images=image, padding=\"max_length\", return_tensors=\"pt\")\n>>> inputs.to(device)\n\n>>> with torch.no_grad():\n...     with torch.autocast(device):\n...         outputs = model(**inputs)\n\n>>> logits_per_image = outputs.logits_per_image\n>>> probs = torch.sigmoid(logits_per_image) # these are the probabilities\n>>> print(f\"{probs[0][0]:.1%} that image 0 is '{candidate_labels[0]}'\")\n51.3% that image 0 is 'This is a photo of 2 cats.'\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 927ebaf6-4d71-47a0-a1fb-058f4d876f3d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"927ebaf6-4d71-47a0-a1fb-058f4d876f3d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"deacb9c7-75ab-464d-adff-6f38f7931701\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4306c05c-a5bb-4668-8d4c-af45d9335822\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0d6885281b80b90be9fabf8b5bbac91e30556baa2e844a7710d01d669c18c5b8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7144, \"end_char_idx\": 9400, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "ref_doc_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "chroma:document": "## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nYou may set `attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used. Make sure you have `torch>=2.1.1`.\n\n```python\n>>> from transformers import SiglipModel\n\n>>> model = SiglipModel.from_pretrained(\n...     \"google/siglip-so400m-patch14-384\",\n...     attn_implementation=\"sdpa\",\n...     torch_dtype=torch.float16,\n...     device_map=device,\n... )\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\n\n## Expected speedups\n\nBelow is an expected speedup diagram that compares inference time between the native implementation in transformers using `google/siglip-so400m-patch14-384` checkpoint in `float16` precision and the Flash Attention 2 / SDPA version of the model using different batch sizes.\n\n<div style=\"text-align: center\">\n<img src=\"https://i.imgur.com/cWm4rsn.png\">\n</div>\n\n\n## SiglipConfig\n\n[[autodoc]] SiglipConfig\n    - from_text_vision_configs\n\n## SiglipTextConfig\n\n[[autodoc]] SiglipTextConfig\n\n## SiglipVisionConfig\n\n[[autodoc]] SiglipVisionConfig\n\n## SiglipTokenizer\n\n[[autodoc]] SiglipTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## SiglipImageProcessor\n\n[[autodoc]] SiglipImageProcessor\n    - preprocess\n\n## SiglipProcessor\n\n[[autodoc]] SiglipProcessor\n\n## SiglipModel\n\n[[autodoc]] SiglipModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## SiglipTextModel\n\n[[autodoc]] SiglipTextModel\n    - forward\n\n## SiglipVisionModel\n\n[[autodoc]] SiglipVisionModel\n    - forward\n\n\n## SiglipForImageClassification\n\n[[autodoc]] SiglipForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: da911c9c-785d-474a-8275-aea72417da22
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/plbart", "title": "PLBart", "tokens": 1313, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"da911c9c-785d-474a-8275-aea72417da22\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b2017e88fe0136f82e097e15171a99d7f9987bc6c4222552a59a4da36795544d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1e31b402-b7eb-4517-9b60-c0e6e20e1dfb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b248ac5d29fed31524e38895ab8ecff7c23812b40791f4125c134ad413f249cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3357, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd", "doc_id": "59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd", "ref_doc_id": "59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd", "chroma:document": "# PLBart\n\n## Overview\n\nThe PLBART model was proposed in [Unified Pre-training for Program Understanding and Generation](https://arxiv.org/abs/2103.06333) by Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang.\nThis is a BART-like model which can be used to perform code-summarization, code-generation, and code-translation tasks. The pre-trained model `plbart-base` has been trained using multilingual denoising task\non Java, Python and English.\n\nAccording to the abstract\n\n*Code summarization and generation empower conversion between programming language (PL) and natural language (NL),\nwhile code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, \na sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks.\nPLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding.\nExperiments on code summarization in the English language, code generation, and code translation in seven programming languages\nshow that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program\nrepair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding.\nFurthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow\n(e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels\neven with limited annotations.*\n\nThis model was contributed by [gchhablani](https://huggingface.co/gchhablani). The Authors' code can be found [here](https://github.com/wasiahmad/PLBART).\n\n## Usage examples\n\nPLBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for code-to-text, text-to-code, code-to-code tasks. As the\nmodel is multilingual it expects the sequences in a different format. A special language id token is added in both the\nsource and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The\ntarget text format is `[tgt_lang_code] X [eos]`. `bos` is never used.\n\nHowever, for fine-tuning, in some cases no language token is provided in cases where a single language is used. Please refer to [the paper](https://arxiv.org/abs/2103.06333) to learn more about this.\n\nIn cases where the language code is needed, the regular [`~PLBartTokenizer.__call__`] will encode source text format \nwhen you pass texts as the first argument or with the keyword argument `text`, and will encode target text format if\nit's passed with the `text_target` keyword argument.\n\n### Supervised training\n\n```python\n>>> from transformers import PLBartForConditionalGeneration, PLBartTokenizer\n\n>>> tokenizer = PLBartTokenizer.from_pretrained(\"uclanlp/plbart-base\", src_lang=\"en_XX\", tgt_lang=\"python\")\n>>> example_python_phrase = \"def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])\"\n>>> expected_translation_english = \"Returns the maximum value of a b c.\"\n>>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors=\"pt\")\n>>> model(**inputs)\n```\n\n### Generation\n\n  While generating the target text set the `decoder_start_token_id` to the target language id."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1e31b402-b7eb-4517-9b60-c0e6e20e1dfb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/plbart", "title": "PLBart", "tokens": 1313, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1e31b402-b7eb-4517-9b60-c0e6e20e1dfb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b2017e88fe0136f82e097e15171a99d7f9987bc6c4222552a59a4da36795544d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da911c9c-785d-474a-8275-aea72417da22\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"60c88eaaf2404eb53eb93210f3cc90abc5851d84628bb88d7fbe71116090800c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1779, \"end_char_idx\": 4835, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd", "doc_id": "59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd", "ref_doc_id": "59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd", "chroma:document": "## Usage examples\n\nPLBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for code-to-text, text-to-code, code-to-code tasks. As the\nmodel is multilingual it expects the sequences in a different format. A special language id token is added in both the\nsource and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The\ntarget text format is `[tgt_lang_code] X [eos]`. `bos` is never used.\n\nHowever, for fine-tuning, in some cases no language token is provided in cases where a single language is used. Please refer to [the paper](https://arxiv.org/abs/2103.06333) to learn more about this.\n\nIn cases where the language code is needed, the regular [`~PLBartTokenizer.__call__`] will encode source text format \nwhen you pass texts as the first argument or with the keyword argument `text`, and will encode target text format if\nit's passed with the `text_target` keyword argument.\n\n### Supervised training\n\n```python\n>>> from transformers import PLBartForConditionalGeneration, PLBartTokenizer\n\n>>> tokenizer = PLBartTokenizer.from_pretrained(\"uclanlp/plbart-base\", src_lang=\"en_XX\", tgt_lang=\"python\")\n>>> example_python_phrase = \"def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])\"\n>>> expected_translation_english = \"Returns the maximum value of a b c.\"\n>>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors=\"pt\")\n>>> model(**inputs)\n```\n\n### Generation\n\n  While generating the target text set the `decoder_start_token_id` to the target language id. The following\n  example shows how to translate Python to English using the `uclanlp/plbart-python-en_XX` model.\n\n```python\n>>> from transformers import PLBartForConditionalGeneration, PLBartTokenizer\n\n>>> tokenizer = PLBartTokenizer.from_pretrained(\"uclanlp/plbart-python-en_XX\", src_lang=\"python\", tgt_lang=\"en_XX\")\n>>> example_python_phrase = \"def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])\"\n>>> inputs = tokenizer(example_python_phrase, return_tensors=\"pt\")\n>>> model = PLBartForConditionalGeneration.from_pretrained(\"uclanlp/plbart-python-en_XX\")\n>>> translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id[\"en_XX\"])\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"Returns the maximum value of a b c.\"\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## PLBartConfig\n\n[[autodoc]] PLBartConfig\n\n## PLBartTokenizer\n\n[[autodoc]] PLBartTokenizer\n    - build_inputs_with_special_tokens\n\n## PLBartModel\n\n[[autodoc]] PLBartModel\n    - forward\n\n## PLBartForConditionalGeneration\n\n[[autodoc]] PLBartForConditionalGeneration\n    - forward\n\n## PLBartForSequenceClassification\n\n[[autodoc]] PLBartForSequenceClassification\n    - forward\n\n## PLBartForCausalLM\n\n[[autodoc]] PLBartForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5ebe8848-4094-444a-84dd-d67fa052aedb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5ebe8848-4094-444a-84dd-d67fa052aedb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"501305bc-fcf7-4429-86e4-3d1e8c07dc20\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f68922bd49394cfb3f1e7866e006558355b27bedf0eb11e95c4a9d3a9e21ab15\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3290, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "# T5\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=t5\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-t5-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/t5-base\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n<a href=\"https://huggingface.co/papers/1910.10683\">\n<img alt=\"Paper page\" src=\"https://img.shields.io/badge/Paper%20page-1910.10683-green\">\n</a>\n</div>\n\n## Overview\n\nThe T5 model was presented in [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf) by [Colin Raffel](https://huggingface.co/craffel), Noam Shazeer, [Adam Roberts](https://huggingface.co/adarob), Katherine Lee, Sharan Narang,\nMichael Matena, Yanqi Zhou, Wei Li, [Peter J. Liu](https://huggingface.co/peterjliu).\n\nThe abstract from the paper is the following:\n\n*Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream\ntask, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning\nhas given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of\ntransfer learning techniques for NLP by introducing a unified framework that converts every language problem into a\ntext-to-text format. Our systematic study compares pretraining objectives, architectures, unlabeled datasets, transfer\napproaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration\nwith scale and our new \"Colossal Clean Crawled Corpus\", we achieve state-of-the-art results on many benchmarks covering\nsummarization, question answering, text classification, and more. To facilitate future work on transfer learning for\nNLP, we release our dataset, pre-trained models, and code.*\n\nAll checkpoints can be found on the [hub](https://huggingface.co/models?search=t5).\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/text-to-text-transfer-transformer).\n\n## Usage tips\n\n- T5 is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised and supervised tasks and for which\neach task is converted into a text-to-text format. T5 works well on a variety of tasks out-of-the-box by prepending a\ndifferent prefix to the input corresponding to each task, e.g., for translation: *translate English to German: ...*,\nfor summarization: *summarize: ...*.\n- The pretraining includes both supervised and self-supervised training. Supervised training is conducted on downstream tasks provided by the GLUE and SuperGLUE benchmarks (converting them into text-to-text tasks as explained above).\n- Self-supervised training uses corrupted tokens, by randomly removing 15% of the tokens and replacing them with individual sentinel tokens (if several consecutive tokens are marked for removal, the whole group is replaced with a single sentinel token). The input of the encoder is the corrupted sentence, the input of the decoder is the original sentence and the target is then the dropped out tokens delimited by their sentinel tokens."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 501305bc-fcf7-4429-86e4-3d1e8c07dc20
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"501305bc-fcf7-4429-86e4-3d1e8c07dc20\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5ebe8848-4094-444a-84dd-d67fa052aedb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9fbdcb72acc43d1bec68a88d3dda44b8cc2548e81b864558560ac3ba23399ab0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ba0eeb09-a79c-4f22-aa22-a3f5db811116\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52522f4920ba72cef3a57155961d36f56a76c52177fd6d68f05ba5a22a66927d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1426, \"end_char_idx\": 4545, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "Our systematic study compares pretraining objectives, architectures, unlabeled datasets, transfer\napproaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration\nwith scale and our new \"Colossal Clean Crawled Corpus\", we achieve state-of-the-art results on many benchmarks covering\nsummarization, question answering, text classification, and more. To facilitate future work on transfer learning for\nNLP, we release our dataset, pre-trained models, and code.*\n\nAll checkpoints can be found on the [hub](https://huggingface.co/models?search=t5).\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/text-to-text-transfer-transformer).\n\n## Usage tips\n\n- T5 is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised and supervised tasks and for which\neach task is converted into a text-to-text format. T5 works well on a variety of tasks out-of-the-box by prepending a\ndifferent prefix to the input corresponding to each task, e.g., for translation: *translate English to German: ...*,\nfor summarization: *summarize: ...*.\n- The pretraining includes both supervised and self-supervised training. Supervised training is conducted on downstream tasks provided by the GLUE and SuperGLUE benchmarks (converting them into text-to-text tasks as explained above).\n- Self-supervised training uses corrupted tokens, by randomly removing 15% of the tokens and replacing them with individual sentinel tokens (if several consecutive tokens are marked for removal, the whole group is replaced with a single sentinel token). The input of the encoder is the corrupted sentence, the input of the decoder is the original sentence and the target is then the dropped out tokens delimited by their sentinel tokens.\n\n- T5 uses relative scalar embeddings. Encoder input padding can be done on the left and on the right.\n\n- See the [training](#training), [inference](#inference) and [resources](#resources) sections below for all details regarding usage.\n\nT5 comes in different sizes:\n\n- [google-t5/t5-small](https://huggingface.co/google-t5/t5-small)\n\n- [google-t5/t5-base](https://huggingface.co/google-t5/t5-base)\n\n- [google-t5/t5-large](https://huggingface.co/google-t5/t5-large)\n\n- [google-t5/t5-3b](https://huggingface.co/google-t5/t5-3b)\n\n- [google-t5/t5-11b](https://huggingface.co/google-t5/t5-11b).\n\nBased on the original T5 model, Google has released some follow-up works:\n\n- **T5v1.1**: T5v1.1 is an improved version of T5 with some architectural tweaks, and is pre-trained on C4 only without\n  mixing in the supervised tasks. Refer to the documentation of T5v1.1 which can be found [here](t5v1.1).\n\n- **mT5**: mT5 is a multilingual T5 model. It is pre-trained on the mC4 corpus, which includes 101 languages. Refer to\n  the documentation of mT5 which can be found [here](mt5).\n\n- **byT5**: byT5 is a T5 model pre-trained on byte sequences rather than SentencePiece subword token sequences. Refer\n  to the documentation of byT5 which can be found [here](byt5)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ba0eeb09-a79c-4f22-aa22-a3f5db811116
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ba0eeb09-a79c-4f22-aa22-a3f5db811116\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"501305bc-fcf7-4429-86e4-3d1e8c07dc20\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3846021b12366a62c285cee3e21ea0d6b6fdcab1c820793363b6dc450b2df969\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cfdc9111-af57-49d3-98c8-ddeb5b325a56\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"54c4a5bfc4c3ccebb41f3ad9d8bb7771056809f593ba47b7258e46411c9bff66\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3292, \"end_char_idx\": 6094, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "- T5 uses relative scalar embeddings. Encoder input padding can be done on the left and on the right.\n\n- See the [training](#training), [inference](#inference) and [resources](#resources) sections below for all details regarding usage.\n\nT5 comes in different sizes:\n\n- [google-t5/t5-small](https://huggingface.co/google-t5/t5-small)\n\n- [google-t5/t5-base](https://huggingface.co/google-t5/t5-base)\n\n- [google-t5/t5-large](https://huggingface.co/google-t5/t5-large)\n\n- [google-t5/t5-3b](https://huggingface.co/google-t5/t5-3b)\n\n- [google-t5/t5-11b](https://huggingface.co/google-t5/t5-11b).\n\nBased on the original T5 model, Google has released some follow-up works:\n\n- **T5v1.1**: T5v1.1 is an improved version of T5 with some architectural tweaks, and is pre-trained on C4 only without\n  mixing in the supervised tasks. Refer to the documentation of T5v1.1 which can be found [here](t5v1.1).\n\n- **mT5**: mT5 is a multilingual T5 model. It is pre-trained on the mC4 corpus, which includes 101 languages. Refer to\n  the documentation of mT5 which can be found [here](mt5).\n\n- **byT5**: byT5 is a T5 model pre-trained on byte sequences rather than SentencePiece subword token sequences. Refer\n  to the documentation of byT5 which can be found [here](byt5).\n\n- **UL2**: UL2 is a T5 like model pretrained on various denoising objectives\n\n- **Flan-T5**: Flan is a pretraining methods that is based on prompting. The Flan-T5 are T5 models trained on the Flan collection of \n    datasets which include: `taskmaster2`, `djaym7/wiki_dialog`, `deepmind/code_contests`, `lambada`, `gsm8k`, `aqua_rat`, `esnli`, `quasc` and `qed`.\n\n- **FLan-UL2** : the UL2 model finetuned using the \"Flan\" prompt tuning and dataset collection.\n\n- **UMT5**: UmT5 is a multilingual T5 model trained on an improved and refreshed mC4 multilingual corpus,  29 trillion characters across 107 language, using a new sampling method, UniMax. Refer to\n the documentation of mT5 which can be found [here](umt5).\n\n## Training\n\nT5 is an encoder-decoder model and converts all NLP problems into a text-to-text format. It is trained using teacher\nforcing. This means that for training, we always need an input sequence and a corresponding target sequence. The input\nsequence is fed to the model using `input_ids`. The target sequence is shifted to the right, i.e., prepended by a\nstart-sequence token and fed to the decoder using the `decoder_input_ids`. In teacher-forcing style, the target\nsequence is then appended by the EOS token and corresponds to the `labels`. The PAD token is hereby used as the\nstart-sequence token. T5 can be trained / fine-tuned both in a supervised and unsupervised fashion.\n\nOne can use [`T5ForConditionalGeneration`] (or the Tensorflow/Flax variant), which includes the\nlanguage modeling head on top of the decoder."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cfdc9111-af57-49d3-98c8-ddeb5b325a56
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cfdc9111-af57-49d3-98c8-ddeb5b325a56\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ba0eeb09-a79c-4f22-aa22-a3f5db811116\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"21381c074555690cf11e8a1785dbc8de89fdcc2b4abe0820584b64db48a607d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1b1de9d0-5e2d-4769-a4cf-f4e6066a258b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bb633e13c4f3b9c1b17eecb5489dcd6ff77b3c3b9b0d9403e8a375645edbd533\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4698, \"end_char_idx\": 7660, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "The Flan-T5 are T5 models trained on the Flan collection of \n    datasets which include: `taskmaster2`, `djaym7/wiki_dialog`, `deepmind/code_contests`, `lambada`, `gsm8k`, `aqua_rat`, `esnli`, `quasc` and `qed`.\n\n- **FLan-UL2** : the UL2 model finetuned using the \"Flan\" prompt tuning and dataset collection.\n\n- **UMT5**: UmT5 is a multilingual T5 model trained on an improved and refreshed mC4 multilingual corpus,  29 trillion characters across 107 language, using a new sampling method, UniMax. Refer to\n the documentation of mT5 which can be found [here](umt5).\n\n## Training\n\nT5 is an encoder-decoder model and converts all NLP problems into a text-to-text format. It is trained using teacher\nforcing. This means that for training, we always need an input sequence and a corresponding target sequence. The input\nsequence is fed to the model using `input_ids`. The target sequence is shifted to the right, i.e., prepended by a\nstart-sequence token and fed to the decoder using the `decoder_input_ids`. In teacher-forcing style, the target\nsequence is then appended by the EOS token and corresponds to the `labels`. The PAD token is hereby used as the\nstart-sequence token. T5 can be trained / fine-tuned both in a supervised and unsupervised fashion.\n\nOne can use [`T5ForConditionalGeneration`] (or the Tensorflow/Flax variant), which includes the\nlanguage modeling head on top of the decoder.\n\n- Unsupervised denoising training\n\nIn this setup, spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens) and\nthe output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens. Each\nsentinel token represents a unique mask token for this sentence and should start with `<extra_id_0>`,\n`<extra_id_1>`, ... up to `<extra_id_99>`. As a default, 100 sentinel tokens are available in\n[`T5Tokenizer`].\n\nFor instance, the sentence \"The cute dog walks in the park\" with the masks put on \"cute dog\" and \"the\" should be\nprocessed as follows:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"The <extra_id_0> walks in <extra_id_1> park\", return_tensors=\"pt\").input_ids\n>>> labels = tokenizer(\"<extra_id_0> cute dog <extra_id_1> the <extra_id_2>\", return_tensors=\"pt\").input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n>>> loss.item()\n3.7837\n```\n\nIf you're interested in pre-training T5 on a new corpus, check out the [run_t5_mlm_flax.py](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling) script in the Examples\ndirectory.\n\n- Supervised training\n\nIn this setup, the input sequence and output sequence are a standard sequence-to-sequence input-output mapping."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1b1de9d0-5e2d-4769-a4cf-f4e6066a258b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1b1de9d0-5e2d-4769-a4cf-f4e6066a258b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cfdc9111-af57-49d3-98c8-ddeb5b325a56\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ce295dc9dfd08ef08f85f5cdfa8b992c81c4f730ca9cf33d55625b28cf3fa0ad\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c9eb92c0-5e5a-4bfe-861f-4267249604cf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a8a16971c0d7f35c23d0d23f4ec6274421fe3c2d777b62ef0e1559313e9a40b5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6096, \"end_char_idx\": 9299, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "- Unsupervised denoising training\n\nIn this setup, spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens) and\nthe output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens. Each\nsentinel token represents a unique mask token for this sentence and should start with `<extra_id_0>`,\n`<extra_id_1>`, ... up to `<extra_id_99>`. As a default, 100 sentinel tokens are available in\n[`T5Tokenizer`].\n\nFor instance, the sentence \"The cute dog walks in the park\" with the masks put on \"cute dog\" and \"the\" should be\nprocessed as follows:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"The <extra_id_0> walks in <extra_id_1> park\", return_tensors=\"pt\").input_ids\n>>> labels = tokenizer(\"<extra_id_0> cute dog <extra_id_1> the <extra_id_2>\", return_tensors=\"pt\").input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n>>> loss.item()\n3.7837\n```\n\nIf you're interested in pre-training T5 on a new corpus, check out the [run_t5_mlm_flax.py](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling) script in the Examples\ndirectory.\n\n- Supervised training\n\nIn this setup, the input sequence and output sequence are a standard sequence-to-sequence input-output mapping.\nSuppose that we want to fine-tune the model for translation for example, and we have a training example: the input\nsequence \"The house is wonderful.\" and output sequence \"Das Haus ist wunderbar.\", then they should be prepared for\nthe model as follows:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"translate English to German: The house is wonderful.\", return_tensors=\"pt\").input_ids\n>>> labels = tokenizer(\"Das Haus ist wunderbar.\", return_tensors=\"pt\").input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n>>> loss.item()\n0.2542\n```\n\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the\n`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded\ntarget sequence). The model will automatically create the `decoder_input_ids` based on the `labels`, by\nshifting them one position to the right and prepending the `config.decoder_start_token_id`, which for T5 is\nequal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate\nEnglish to German: ' before encoding it. This will help in improving the performance, as this task prefix was used\nduring T5's pre-training.\n\nHowever, the example above only shows a single training example. In practice, one trains deep learning models in\nbatches."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c9eb92c0-5e5a-4bfe-861f-4267249604cf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c9eb92c0-5e5a-4bfe-861f-4267249604cf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1b1de9d0-5e2d-4769-a4cf-f4e6066a258b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15fab9815d1ab9f894595f1ba42cc0f038638e3bba37c025c34942a1c9c60f89\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3fd70fc2-d5dd-4931-b015-f9309ffc9584\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e6b69ae207503c9dc0da54c984ced6591351883a4949717d29a4acfeb09bea86\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7661, \"end_char_idx\": 10278, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "Suppose that we want to fine-tune the model for translation for example, and we have a training example: the input\nsequence \"The house is wonderful.\" and output sequence \"Das Haus ist wunderbar.\", then they should be prepared for\nthe model as follows:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"translate English to German: The house is wonderful.\", return_tensors=\"pt\").input_ids\n>>> labels = tokenizer(\"Das Haus ist wunderbar.\", return_tensors=\"pt\").input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n>>> loss.item()\n0.2542\n```\n\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the\n`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded\ntarget sequence). The model will automatically create the `decoder_input_ids` based on the `labels`, by\nshifting them one position to the right and prepending the `config.decoder_start_token_id`, which for T5 is\nequal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate\nEnglish to German: ' before encoding it. This will help in improving the performance, as this task prefix was used\nduring T5's pre-training.\n\nHowever, the example above only shows a single training example. In practice, one trains deep learning models in\nbatches. This entails that we must pad/truncate examples to the same length. For encoder-decoder models, one\ntypically defines a `max_source_length` and `max_target_length`, which determine the maximum length of the\ninput and output sequences respectively (otherwise they are truncated). These should be carefully set depending on\nthe task.\n\nIn addition, we must make sure that padding token id's of the `labels` are not taken into account by the loss\nfunction. In PyTorch and Tensorflow, this can be done by replacing them with -100, which is the `ignore_index`\nof the `CrossEntropyLoss`. In Flax, one can use the `decoder_attention_mask` to ignore padded tokens from\nthe loss (see the [Flax summarization script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization) for details). We also pass\n`attention_mask` as additional input to the model, which makes sure that padding tokens of the inputs are\nignored. The code example below illustrates all of this."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3fd70fc2-d5dd-4931-b015-f9309ffc9584
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3fd70fc2-d5dd-4931-b015-f9309ffc9584\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c9eb92c0-5e5a-4bfe-861f-4267249604cf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a9c0fcf18b2caac0ea1892e5c9870c6e88e5d912ca4b0f51e3211149bc171e71\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"582bc3c4-c231-43c1-a404-0eccb803b3db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bd35e6532e00f9a90d7557771087ff9ccfef3215765ffad0a31f4965c238966d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8726, \"end_char_idx\": 12002, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "The model will automatically create the `decoder_input_ids` based on the `labels`, by\nshifting them one position to the right and prepending the `config.decoder_start_token_id`, which for T5 is\nequal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate\nEnglish to German: ' before encoding it. This will help in improving the performance, as this task prefix was used\nduring T5's pre-training.\n\nHowever, the example above only shows a single training example. In practice, one trains deep learning models in\nbatches. This entails that we must pad/truncate examples to the same length. For encoder-decoder models, one\ntypically defines a `max_source_length` and `max_target_length`, which determine the maximum length of the\ninput and output sequences respectively (otherwise they are truncated). These should be carefully set depending on\nthe task.\n\nIn addition, we must make sure that padding token id's of the `labels` are not taken into account by the loss\nfunction. In PyTorch and Tensorflow, this can be done by replacing them with -100, which is the `ignore_index`\nof the `CrossEntropyLoss`. In Flax, one can use the `decoder_attention_mask` to ignore padded tokens from\nthe loss (see the [Flax summarization script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization) for details). We also pass\n`attention_mask` as additional input to the model, which makes sure that padding tokens of the inputs are\nignored. The code example below illustrates all of this.\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n>>> import torch\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> # the following 2 hyperparameters are task-specific\n>>> max_source_length = 512\n>>> max_target_length = 128\n\n>>> # Suppose we have the following 2 training examples:\n>>> input_sequence_1 = \"Welcome to NYC\"\n>>> output_sequence_1 = \"Bienvenue \u00e0 NYC\"\n\n>>> input_sequence_2 = \"HuggingFace is a company\"\n>>> output_sequence_2 = \"HuggingFace est une entreprise\"\n\n>>> # encode the inputs\n>>> task_prefix = \"translate English to French: \"\n>>> input_sequences = [input_sequence_1, input_sequence_2]\n\n>>> encoding = tokenizer(\n...     [task_prefix + sequence for sequence in input_sequences],\n...     padding=\"longest\",\n...     max_length=max_source_length,\n...     truncation=True,\n...     return_tensors=\"pt\",\n... )\n\n>>> input_ids, attention_mask = encoding.input_ids, encoding.attention_mask\n\n>>> # encode the targets\n>>> target_encoding = tokenizer(\n...     [output_sequence_1, output_sequence_2],\n...     padding=\"longest\",\n...     max_length=max_target_length,\n...     truncation=True,\n...     return_tensors=\"pt\",\n... )\n>>> labels = target_encoding.input_ids\n\n>>> # replace padding token id's of the labels by -100 so it's ignored by the loss\n>>> labels[labels == tokenizer.pad_token_id] = -100\n\n>>> # forward pass\n>>> loss = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels).loss\n>>> loss.item()\n0.188\n```\n\nAdditional training tips:\n\n- T5 models need a slightly higher learning rate than the default one set in the `Trainer` when using the AdamW\noptimizer."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 582bc3c4-c231-43c1-a404-0eccb803b3db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"582bc3c4-c231-43c1-a404-0eccb803b3db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3fd70fc2-d5dd-4931-b015-f9309ffc9584\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"318027b0eacb926bf790535c00ae848d0aacc9927eda43669d7d69f6b35244b6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23eb8394-e627-4cef-9f6b-a4dae5d7484b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f8b8aca254aa0e29bab97502b7aa35174c9b86ccb4aff1c7038476cf1c460d4c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12003, \"end_char_idx\": 15403, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "Typically, 1e-4 and 3e-4 work well for most problems (classification, summarization, translation, question\nanswering, question generation). Note that T5 was pre-trained using the AdaFactor optimizer.\n\nAccording to [this forum post](https://discuss.huggingface.co/t/t5-finetuning-tips/684), task prefixes matter when\n(1) doing multi-task training (2) your task is similar or related to one of the supervised tasks used in T5's\npre-training mixture (see Appendix D of the [paper](https://arxiv.org/pdf/1910.10683.pdf) for the task prefixes\nused).\n\nIf training on TPU, it is recommended to pad all examples of the dataset to the same length or make use of\n*pad_to_multiple_of* to have a small number of predefined bucket sizes to fit all examples in. Dynamically padding\nbatches to the longest example is not recommended on TPU as it triggers a recompilation for every batch shape that is\nencountered during training thus significantly slowing down the training. only padding up to the longest example in a\nbatch) leads to very slow training on TPU.\n\n## Inference\n\nAt inference time, it is recommended to use [`~generation.GenerationMixin.generate`]. This\nmethod takes care of encoding the input and feeding the encoded hidden states via cross-attention layers to the decoder\nand auto-regressively generates the decoder output. Check out [this blog post](https://huggingface.co/blog/how-to-generate) to know all the details about generating text with Transformers.\nThere's also [this blog post](https://huggingface.co/blog/encoder-decoder#encoder-decoder) which explains how\ngeneration works in general in encoder-decoder models.\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"translate English to German: The house is wonderful.\", return_tensors=\"pt\").input_ids\n>>> outputs = model.generate(input_ids)\n>>> print(tokenizer.decode(outputs[0], skip_special_tokens=True))\nDas Haus ist wunderbar.\n```\n\nNote that T5 uses the `pad_token_id` as the `decoder_start_token_id`, so when doing generation without using\n[`~generation.GenerationMixin.generate`], make sure you start it with the `pad_token_id`.\n\nThe example above only shows a single example. You can also do batched inference, like so:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> task_prefix = \"translate English to German: \"\n>>> # use different length sentences to test batching\n>>> sentences = [\"The house is wonderful.\", \"I like to work in NYC.\"]\n\n>>> inputs = tokenizer([task_prefix + sentence for sentence in sentences], return_tensors=\"pt\", padding=True)\n\n>>> output_sequences = model.generate(\n...     input_ids=inputs[\"input_ids\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     do_sample=False,  # disable sampling to test if batching affects output\n... )\n\n>>> print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))\n['Das Haus ist wunderbar.', 'Ich arbeite gerne in NYC.']\n```\n\nBecause T5 has been trained with the span-mask denoising objective,\nit can be used to predict the sentinel (masked-out) tokens during inference."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 23eb8394-e627-4cef-9f6b-a4dae5d7484b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"23eb8394-e627-4cef-9f6b-a4dae5d7484b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"582bc3c4-c231-43c1-a404-0eccb803b3db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a9bcb5e14a8a2f5ccbe9696883824a5e157e3a9f3eb16260c36b5d91bb9a51fb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e5fe22a1-52e6-46af-a8e1-6359dd20b96a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aaadb9ca0ec2d5b4fa1e4c683e430ca143106d255ebdf553d31d09019a2b6a53\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13934, \"end_char_idx\": 17180, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "\", return_tensors=\"pt\").input_ids\n>>> outputs = model.generate(input_ids)\n>>> print(tokenizer.decode(outputs[0], skip_special_tokens=True))\nDas Haus ist wunderbar.\n```\n\nNote that T5 uses the `pad_token_id` as the `decoder_start_token_id`, so when doing generation without using\n[`~generation.GenerationMixin.generate`], make sure you start it with the `pad_token_id`.\n\nThe example above only shows a single example. You can also do batched inference, like so:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> task_prefix = \"translate English to German: \"\n>>> # use different length sentences to test batching\n>>> sentences = [\"The house is wonderful.\", \"I like to work in NYC.\"]\n\n>>> inputs = tokenizer([task_prefix + sentence for sentence in sentences], return_tensors=\"pt\", padding=True)\n\n>>> output_sequences = model.generate(\n...     input_ids=inputs[\"input_ids\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     do_sample=False,  # disable sampling to test if batching affects output\n... )\n\n>>> print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))\n['Das Haus ist wunderbar.', 'Ich arbeite gerne in NYC.']\n```\n\nBecause T5 has been trained with the span-mask denoising objective,\nit can be used to predict the sentinel (masked-out) tokens during inference.\nThe predicted tokens will then be placed between the sentinel tokens.\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"The <extra_id_0> walks in <extra_id_1> park\", return_tensors=\"pt\").input_ids\n\n>>> sequence_ids = model.generate(input_ids)\n>>> sequences = tokenizer.batch_decode(sequence_ids)\n>>> sequences\n['<pad> <extra_id_0> park offers <extra_id_1> the <extra_id_2> park.</s>']\n```\n\n## Performance\n\nIf you'd like a faster training and inference performance, install [NVIDIA APEX](https://github.com/NVIDIA/apex#quick-start) for NVIDIA GPUs, or [ROCm APEX](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs and then the model will automatically use `apex.normalization.FusedRMSNorm` instead of `T5LayerNorm`. The former uses an optimized fused kernel which is several times faster than the latter.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with T5. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A notebook for how to [finetune T5 for classification and multiple choice](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb).\n- A notebook for how to [finetune T5 for sentiment span extraction](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e5fe22a1-52e6-46af-a8e1-6359dd20b96a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e5fe22a1-52e6-46af-a8e1-6359dd20b96a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"23eb8394-e627-4cef-9f6b-a4dae5d7484b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e611367013c76f92267a4b6d9c51677bcf242c2fcd9f20bc7a549ecde5437e6f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f0abd6ae-9a70-49bb-9811-1aaab8e8f6fb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e0846fb07b468242044b588834cf5575d8e8886a8a335e2b73ce5b974ccfd3cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16330, \"end_char_idx\": 19274, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "The former uses an optimized fused kernel which is several times faster than the latter.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with T5. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A notebook for how to [finetune T5 for classification and multiple choice](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb).\n- A notebook for how to [finetune T5 for sentiment span extraction](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb). \ud83c\udf0e\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A notebook for how to [finetune T5 for named entity recognition](https://colab.research.google.com/drive/1obr78FY_cBmWY5ODViCmzdY6O1KB65Vc?usp=sharing). \ud83c\udf0e\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A notebook for [Finetuning CodeT5 for generating docstrings from Ruby code](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/T5/Fine_tune_CodeT5_for_generating_docstrings_from_Ruby_code.ipynb).\n\n<PipelineTag pipeline=\"summarization\"/>\n\n- A notebook to [Finetune T5-base-dutch to perform Dutch abstractive summarization on a TPU](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/T5/Fine_tuning_Dutch_T5_base_on_CNN_Daily_Mail_for_summarization_(on_TPU_using_HuggingFace_Accelerate).ipynb).\n- A notebook for how to [finetune T5 for summarization in PyTorch and track experiments with WandB](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb#scrollTo=OKRpFvYhBauC). \ud83c\udf0e\n- A blog post on [Distributed Training: Train BART/T5 for Summarization using \ud83e\udd17 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).\n- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).\n- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n- [`FlaxT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).\n- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the \ud83e\udd17 Hugging Face course."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f0abd6ae-9a70-49bb-9811-1aaab8e8f6fb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f0abd6ae-9a70-49bb-9811-1aaab8e8f6fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e5fe22a1-52e6-46af-a8e1-6359dd20b96a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b7af59cb6076a6d4fad1d83f0bd6b43e14266390bba22b174d67d68287a5408e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"717adf7a-1ca2-4e96-a3b5-4a1dfa69fb33\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7aae1798735c66dba75a40823f3a1a053805c1176ae65f33a962872f835a67b5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18000, \"end_char_idx\": 20960, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "- A notebook for how to [finetune T5 for summarization in PyTorch and track experiments with WandB](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb#scrollTo=OKRpFvYhBauC). \ud83c\udf0e\n- A blog post on [Distributed Training: Train BART/T5 for Summarization using \ud83e\udd17 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).\n- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).\n- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n- [`FlaxT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).\n- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the \ud83e\udd17 Hugging Face course.\n- [Summarization task guide](../tasks/summarization)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`FlaxT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#t5-like-span-masked-language-modeling) for training T5 with a span-masked language model objective. The script also shows how to train a T5 tokenizer. [`FlaxT5ForConditionalGeneration`] is also supported by this [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n\n<PipelineTag pipeline=\"translation\"/>\n\n- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).\n- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n- [Translation task guide](../tasks/translation)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A notebook on how to [finetune T5 for question answering with TensorFlow 2](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune T5 for question answering on a TPU](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 717adf7a-1ca2-4e96-a3b5-4a1dfa69fb33
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"717adf7a-1ca2-4e96-a3b5-4a1dfa69fb33\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1c3fd25-ee81-4007-ade4-89527a28caf2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0abd6ae-9a70-49bb-9811-1aaab8e8f6fb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e6f7d959dc03f25232c1b699e700a631b1c90760839cc88043e83f6d8d7740\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19627, \"end_char_idx\": 22332, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "ref_doc_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "chroma:document": "The script also shows how to train a T5 tokenizer. [`FlaxT5ForConditionalGeneration`] is also supported by this [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n\n<PipelineTag pipeline=\"translation\"/>\n\n- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).\n- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n- [Translation task guide](../tasks/translation)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A notebook on how to [finetune T5 for question answering with TensorFlow 2](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune T5 for question answering on a TPU](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil).\n\n\ud83d\ude80 **Deploy**\n- A blog post on how to deploy [T5 11B for inference for less than $500](https://www.philschmid.de/deploy-t5-11b).\n\n## T5Config\n\n[[autodoc]] T5Config\n\n## T5Tokenizer\n\n[[autodoc]] T5Tokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## T5TokenizerFast\n\n[[autodoc]] T5TokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## T5Model\n\n[[autodoc]] T5Model\n    - forward\n\n## T5ForConditionalGeneration\n\n[[autodoc]] T5ForConditionalGeneration\n    - forward\n\n## T5EncoderModel\n\n[[autodoc]] T5EncoderModel\n    - forward\n\n## T5ForSequenceClassification\n\n[[autodoc]] T5ForSequenceClassification\n    - forward\n\n## T5ForTokenClassification\n\n[[autodoc]] T5ForTokenClassification\n    - forward\n\n## T5ForQuestionAnswering\n\n[[autodoc]] T5ForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFT5Model\n\n[[autodoc]] TFT5Model\n    - call\n\n## TFT5ForConditionalGeneration\n\n[[autodoc]] TFT5ForConditionalGeneration\n    - call\n\n## TFT5EncoderModel\n\n[[autodoc]] TFT5EncoderModel\n    - call\n\n</tf>\n<jax>\n\n## FlaxT5Model\n\n[[autodoc]] FlaxT5Model\n    - __call__\n    - encode\n    - decode\n\n## FlaxT5ForConditionalGeneration\n\n[[autodoc]] FlaxT5ForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n## FlaxT5EncoderModel\n\n[[autodoc]] FlaxT5EncoderModel\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f11fd7dd-6584-4f69-bd95-5a794ee5121d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f11fd7dd-6584-4f69-bd95-5a794ee5121d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23c21ffd-fff2-4ac5-b292-68fd917442c7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5c546cc2f33c219d055b079dd32ff47547c5ee34f803231abcd7426b847fcc0e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2776, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "# OpenAI GPT2\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=gpt2\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-gpt2-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/gpt2\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nOpenAI GPT-2 model was proposed in [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) by Alec\nRadford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever from [OpenAI](https://huggingface.co/openai). It's a causal (unidirectional)\ntransformer pretrained using language modeling on a very large corpus of ~40 GB of text data.\n\nThe abstract from the paper is the following:\n\n*GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset[1] of 8 million\nweb pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some\ntext. The diversity of the dataset causes this simple goal to contain naturally occurring demonstrations of many tasks\nacross diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than\n10X the amount of data.*\n\n[Write With Transformer](https://transformer.huggingface.co/doc/gpt2-large) is a webapp created and hosted by\nHugging Face showcasing the generative capabilities of several models. GPT-2 is one of them and is available in five\ndifferent sizes: small, medium, large, xl and a distilled version of the small checkpoint: *distilgpt-2*.\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://openai.com/blog/better-language-models/).\n\n## Usage tips\n\n- GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next\n  token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be\n  observed in the *run_generation.py* example script.\n- The model can take the *past_key_values* (for PyTorch) or *past* (for TF) as input, which is the previously computed\n  key/value attention pairs. Using this (*past_key_values* or *past*) value prevents the model from re-computing\n  pre-computed values in the context of text generation. For PyTorch, see *past_key_values* argument of the\n  [`GPT2Model.forward`] method, or for TF the *past* argument of the\n  [`TFGPT2Model.call`] method for more information on its usage."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 23c21ffd-fff2-4ac5-b292-68fd917442c7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"23c21ffd-fff2-4ac5-b292-68fd917442c7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f11fd7dd-6584-4f69-bd95-5a794ee5121d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2afcc1e6695848523a447a8fd4aae9c493a3019851468330c043eb0fe0ff51a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5646deeb-9fae-4061-955e-8c8853e558b0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e5fc09560edb7effbd9ab67a333c03eda5b98ed4acbc8cdf0de13133277290d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1254, \"end_char_idx\": 4229, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than\n10X the amount of data.*\n\n[Write With Transformer](https://transformer.huggingface.co/doc/gpt2-large) is a webapp created and hosted by\nHugging Face showcasing the generative capabilities of several models. GPT-2 is one of them and is available in five\ndifferent sizes: small, medium, large, xl and a distilled version of the small checkpoint: *distilgpt-2*.\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://openai.com/blog/better-language-models/).\n\n## Usage tips\n\n- GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next\n  token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be\n  observed in the *run_generation.py* example script.\n- The model can take the *past_key_values* (for PyTorch) or *past* (for TF) as input, which is the previously computed\n  key/value attention pairs. Using this (*past_key_values* or *past*) value prevents the model from re-computing\n  pre-computed values in the context of text generation. For PyTorch, see *past_key_values* argument of the\n  [`GPT2Model.forward`] method, or for TF the *past* argument of the\n  [`TFGPT2Model.call`] method for more information on its usage.\n- Enabling the *scale_attn_by_inverse_layer_idx* and *reorder_and_upcast_attn* flags will apply the training stability\n  improvements from [Mistral](https://github.com/stanford-crfm/mistral/) (for PyTorch only).\n\n## Usage example\n\nThe `generate()` method can be used to generate text using GPT2 model.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n>>> prompt = \"GPT2 is a model developed by OpenAI.\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5646deeb-9fae-4061-955e-8c8853e558b0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5646deeb-9fae-4061-955e-8c8853e558b0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"23c21ffd-fff2-4ac5-b292-68fd917442c7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5d8f4f5ab7558d55de998f8a0c48985380ae7606db0f52835daea5d2e00d3ce4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dfa911f4-cea2-49fd-ad62-ba8dfe82134c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"813428da12c70079b3f413631e3a9d54dd1d9f554818584433187e584e95ae71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2592, \"end_char_idx\": 5876, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "For PyTorch, see *past_key_values* argument of the\n  [`GPT2Model.forward`] method, or for TF the *past* argument of the\n  [`TFGPT2Model.call`] method for more information on its usage.\n- Enabling the *scale_attn_by_inverse_layer_idx* and *reorder_and_upcast_attn* flags will apply the training stability\n  improvements from [Mistral](https://github.com/stanford-crfm/mistral/) (for PyTorch only).\n\n## Usage example\n\nThe `generate()` method can be used to generate text using GPT2 model.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n>>> prompt = \"GPT2 is a model developed by OpenAI.\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"gpt2\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n```\n\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `gpt2` checkpoint and the Flash Attention 2 version of the model using a sequence length of 512.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/EduardoPacheco/documentation-images/resolve/main/gpt2_flash_attention_2_speedup.jpg\">\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dfa911f4-cea2-49fd-ad62-ba8dfe82134c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dfa911f4-cea2-49fd-ad62-ba8dfe82134c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5646deeb-9fae-4061-955e-8c8853e558b0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc49e1c2393ac5bf979cce9ffc9f3e5c253769660a5fd577935b3e962bd495f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"28cd0b23-7ca8-4b6b-a27f-f3f6e133415f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3990f5f95ce583b47a4df6958c2ccdd47cac1ea2db295384b46798e9ea3109bb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4231, \"end_char_idx\": 7135, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"gpt2\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n```\n\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `gpt2` checkpoint and the Flash Attention 2 version of the model using a sequence length of 512.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/EduardoPacheco/documentation-images/resolve/main/gpt2_flash_attention_2_speedup.jpg\">\n</div>\n\n\n## Using Scaled Dot Product Attention (SDPA)\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function\nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the\n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)\nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set\n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import AutoModelForCausalLM\nmodel = AutoModelForCausalLM.from_pretrained(\"gpt2\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with\n[gpt2-large](https://huggingface.co/openai-community/gpt2-large), we saw the\nfollowing speedups during training and inference."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 28cd0b23-7ca8-4b6b-a27f-f3f6e133415f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"28cd0b23-7ca8-4b6b-a27f-f3f6e133415f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dfa911f4-cea2-49fd-ad62-ba8dfe82134c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1f696c2265f8edebd6ec03d5daa62290059acc0f3538eacefe51b3e4e8e84196\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ebfb556f-27a8-49d5-b19d-47f4531ba0d6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aa500fff1de9a4a5a23454332b8ea8924cfa71a69d2a4a1f8e659de5fbdd6c49\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5879, \"end_char_idx\": 8725, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "## Using Scaled Dot Product Attention (SDPA)\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function\nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the\n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)\nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set\n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import AutoModelForCausalLM\nmodel = AutoModelForCausalLM.from_pretrained(\"gpt2\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with\n[gpt2-large](https://huggingface.co/openai-community/gpt2-large), we saw the\nfollowing speedups during training and inference.\n\n### Training\n| Batch size | Seq len |  Time per batch (Eager - s) | Time per batch (SDPA - s) | Speedup (%) | Eager peak mem (MB) | SDPA peak mem (MB) |    Mem saving (%) |\n|-----------:|--------:|----------------------------:|--------------------------:|------------:|--------------------:|-------------------:|------------------:|\n|          1 |     128 |                       0.039 |                     0.032 |      23.042 |             3482.32 |            3494.62 |            -0.352 |\n|          1 |     256 |                       0.073 |                     0.059 |       25.15 |             3546.66 |             3552.6 |            -0.167 |\n|          1 |     512 |                       0.155 |                     0.118 |       30.96 |              4230.1 |            3665.59 |              15.4 |\n|          1 |    1024 |                       0.316 |                     0.209 |      50.839 |             8682.26 |            4881.09 |            77.875 |\n|          2 |     128 |                        0.07 |                      0.06 |      15.324 |              3557.8 |            3545.91 |             0.335 |\n|          2 |     256 |                       0.143 |                     0.122 |       16.53 |              3901.5 |            3657.68 |             6.666 |\n|          2 |     512 |                       0.267 |                     0.213 |      25.626 |             7062.21 |            4876.47 |            44.822 |\n|          2 |    1024 |                         OOM |                     0.404 |           / |                 OOM |            8096."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ebfb556f-27a8-49d5-b19d-47f4531ba0d6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ebfb556f-27a8-49d5-b19d-47f4531ba0d6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"28cd0b23-7ca8-4b6b-a27f-f3f6e133415f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6633f199f3d590e898e23eb0248907889d797e4279053318f60cf03b21725492\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f831ca3-5fcf-41af-a639-0f939efa3942\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"47019343afe82b5dc7d745c78f3acfc0a99e588645e3a0cb6c11a8bebaa7275c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7519, \"end_char_idx\": 10244, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "039 |                     0.032 |      23.042 |             3482.32 |            3494.62 |            -0.352 |\n|          1 |     256 |                       0.073 |                     0.059 |       25.15 |             3546.66 |             3552.6 |            -0.167 |\n|          1 |     512 |                       0.155 |                     0.118 |       30.96 |              4230.1 |            3665.59 |              15.4 |\n|          1 |    1024 |                       0.316 |                     0.209 |      50.839 |             8682.26 |            4881.09 |            77.875 |\n|          2 |     128 |                        0.07 |                      0.06 |      15.324 |              3557.8 |            3545.91 |             0.335 |\n|          2 |     256 |                       0.143 |                     0.122 |       16.53 |              3901.5 |            3657.68 |             6.666 |\n|          2 |     512 |                       0.267 |                     0.213 |      25.626 |             7062.21 |            4876.47 |            44.822 |\n|          2 |    1024 |                         OOM |                     0.404 |           / |                 OOM |            8096.35 | SDPA does not OOM |\n|          4 |     128 |                       0.134 |                     0.128 |       4.412 |             3675.79 |            3648.72 |             0.742 |\n|          4 |     256 |                       0.243 |                     0.217 |      12.292 |             6129.76 |            4871.12 |            25.839 |\n|          4 |     512 |                       0.494 |                     0.406 |      21.687 |             12466.6 |            8102.64 |            53.858 |\n|          4 |    1024 |                         OOM |                     0.795 |           / |                 OOM |            14568.2 | SDPA does not OOM |\n\n### Inference\n| Batch size | Seq len | Per token latency Eager (ms) | Per token latency SDPA (ms) | Speedup (%) | Mem Eager (MB) | Mem SDPA (MB) | Mem saved (%) |\n|-----------:|--------:|-----------------------------:|----------------------------:|------------:|---------------:|--------------:|--------------:|\n|          1 |     128 |                        7.991 |                       6.968 |      14.681 |         1685.2 |       1701.32 |        -0.947 |\n|          1 |     256 |                        8.462 |                       7.199 |      17.536 |        1745.49 |       1770.78 |        -1.428 |\n|          1 |     512 |                         8.68 |                       7.853 |      10.529 |        1907.69 |       1921.29 |        -0.708 |\n|          1 |     768 |                        9.101 |                       8.365 |       8."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3f831ca3-5fcf-41af-a639-0f939efa3942
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3f831ca3-5fcf-41af-a639-0f939efa3942\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ebfb556f-27a8-49d5-b19d-47f4531ba0d6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"05ecf144cd50956cde7813fec81a113c1b6c2677e1c9e7736b3e9989bea146f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0679d3bf-519b-476c-a190-141864333563\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"12acf598121ee8eadf85735bfbbfe4d38748e5fda0f93f832a75e7492c42ed04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8841, \"end_char_idx\": 11454, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "412 |             3675.79 |            3648.72 |             0.742 |\n|          4 |     256 |                       0.243 |                     0.217 |      12.292 |             6129.76 |            4871.12 |            25.839 |\n|          4 |     512 |                       0.494 |                     0.406 |      21.687 |             12466.6 |            8102.64 |            53.858 |\n|          4 |    1024 |                         OOM |                     0.795 |           / |                 OOM |            14568.2 | SDPA does not OOM |\n\n### Inference\n| Batch size | Seq len | Per token latency Eager (ms) | Per token latency SDPA (ms) | Speedup (%) | Mem Eager (MB) | Mem SDPA (MB) | Mem saved (%) |\n|-----------:|--------:|-----------------------------:|----------------------------:|------------:|---------------:|--------------:|--------------:|\n|          1 |     128 |                        7.991 |                       6.968 |      14.681 |         1685.2 |       1701.32 |        -0.947 |\n|          1 |     256 |                        8.462 |                       7.199 |      17.536 |        1745.49 |       1770.78 |        -1.428 |\n|          1 |     512 |                         8.68 |                       7.853 |      10.529 |        1907.69 |       1921.29 |        -0.708 |\n|          1 |     768 |                        9.101 |                       8.365 |       8.791 |        2032.93 |       2068.12 |        -1.701 |\n|          2 |     128 |                        9.169 |                       9.001 |       1.861 |        1803.84 |        1811.4 |        -0.418 |\n|          2 |     256 |                        9.907 |                        9.78 |       1.294 |        1907.72 |       1921.44 |        -0.714 |\n|          2 |     512 |                       11.519 |                      11.644 |      -1.071 |        2176.86 |       2197.75 |        -0.951 |\n|          2 |     768 |                       13.022 |                      13.407 |      -2.873 |         2464.3 |       2491.06 |        -1.074 |\n|          4 |     128 |                       10.097 |                       9.831 |       2.709 |        1942.25 |       1985.13 |         -2.16 |\n|          4 |     256 |                       11.599 |                      11.398 |       1.764 |        2177.28 |       2197.86 |        -0.937 |\n|          4 |     512 |                       14.653 |                       14.45 |       1.411 |        2753.16 |       2772.57 |          -0.7 |\n|          4 |     768 |                       17.846 |                      17.617 |       1.299 |        3327."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0679d3bf-519b-476c-a190-141864333563
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0679d3bf-519b-476c-a190-141864333563\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3f831ca3-5fcf-41af-a639-0f939efa3942\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d5e8ad10cfc9b13e2e2dc6809d5d19edf1a60889269c5d6055c95c421aa8983f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7ce65e59-b06a-441f-8363-a639c7a36906\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c1e143b8950968e7651beafcfef36155e5dc8a158284190cf6846d1c2bb73270\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10200, \"end_char_idx\": 12773, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "101 |                       8.365 |       8.791 |        2032.93 |       2068.12 |        -1.701 |\n|          2 |     128 |                        9.169 |                       9.001 |       1.861 |        1803.84 |        1811.4 |        -0.418 |\n|          2 |     256 |                        9.907 |                        9.78 |       1.294 |        1907.72 |       1921.44 |        -0.714 |\n|          2 |     512 |                       11.519 |                      11.644 |      -1.071 |        2176.86 |       2197.75 |        -0.951 |\n|          2 |     768 |                       13.022 |                      13.407 |      -2.873 |         2464.3 |       2491.06 |        -1.074 |\n|          4 |     128 |                       10.097 |                       9.831 |       2.709 |        1942.25 |       1985.13 |         -2.16 |\n|          4 |     256 |                       11.599 |                      11.398 |       1.764 |        2177.28 |       2197.86 |        -0.937 |\n|          4 |     512 |                       14.653 |                       14.45 |       1.411 |        2753.16 |       2772.57 |          -0.7 |\n|          4 |     768 |                       17.846 |                      17.617 |       1.299 |        3327.04 |       3343.97 |        -0.506 |\n\n\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GPT2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A blog on how to [Finetune a non-English GPT-2 Model with Hugging Face](https://www.philschmid.de/fine-tune-a-non-english-gpt-2-model-with-huggingface).\n- A blog on [How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate) with GPT-2.\n- A blog on [Training CodeParrot \ud83e\udd9c from Scratch](https://huggingface.co/blog/codeparrot), a large GPT-2 model.\n- A blog on [Faster Text Generation with TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate) with GPT-2.\n- A blog on [How to train a Language Model with Megatron-LM](https://huggingface.co/blog/megatron-training) with a GPT-2 model.\n- A notebook on how to [finetune GPT2 to generate lyrics in the style of your favorite artist](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7ce65e59-b06a-441f-8363-a639c7a36906
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7ce65e59-b06a-441f-8363-a639c7a36906\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0679d3bf-519b-476c-a190-141864333563\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c8fa34a85957facdd49f9769cf37a4a519b7f54a5dbb7b4651f3415d60e64a58\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7f42f320-0a3d-463a-a23a-8e781300a661\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ec4de0dc4e01cdfe73437432d4bbe67267687c739604a5f35904b7f8ec1ee7fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11305, \"end_char_idx\": 14258, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "16 |       2772.57 |          -0.7 |\n|          4 |     768 |                       17.846 |                      17.617 |       1.299 |        3327.04 |       3343.97 |        -0.506 |\n\n\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GPT2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A blog on how to [Finetune a non-English GPT-2 Model with Hugging Face](https://www.philschmid.de/fine-tune-a-non-english-gpt-2-model-with-huggingface).\n- A blog on [How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate) with GPT-2.\n- A blog on [Training CodeParrot \ud83e\udd9c from Scratch](https://huggingface.co/blog/codeparrot), a large GPT-2 model.\n- A blog on [Faster Text Generation with TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate) with GPT-2.\n- A blog on [How to train a Language Model with Megatron-LM](https://huggingface.co/blog/megatron-training) with a GPT-2 model.\n- A notebook on how to [finetune GPT2 to generate lyrics in the style of your favorite artist](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). \ud83c\udf0e\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`GPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7f42f320-0a3d-463a-a23a-8e781300a661
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7f42f320-0a3d-463a-a23a-8e781300a661\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a89b4659-8bb4-4786-a637-d4af83954f55\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7ce65e59-b06a-441f-8363-a639c7a36906\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"31a8c05e1b50dbab2ffd6f7bf13fd5daf460ba68bab2286030b39bf750933653\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12774, \"end_char_idx\": 15876, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "ref_doc_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "chroma:document": "\ud83c\udf0e\n- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). \ud83c\udf0e\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`GPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## GPT2Config\n\n[[autodoc]] GPT2Config\n\n## GPT2Tokenizer\n\n[[autodoc]] GPT2Tokenizer\n    - save_vocabulary\n\n## GPT2TokenizerFast\n\n[[autodoc]] GPT2TokenizerFast\n\n## GPT2 specific outputs\n\n[[autodoc]] models.gpt2.modeling_gpt2.GPT2DoubleHeadsModelOutput\n\n[[autodoc]] models.gpt2.modeling_tf_gpt2.TFGPT2DoubleHeadsModelOutput\n\n<frameworkcontent>\n<pt>\n\n## GPT2Model\n\n[[autodoc]] GPT2Model\n    - forward\n\n## GPT2LMHeadModel\n\n[[autodoc]] GPT2LMHeadModel\n    - forward\n\n## GPT2DoubleHeadsModel\n\n[[autodoc]] GPT2DoubleHeadsModel\n    - forward\n\n## GPT2ForQuestionAnswering\n\n[[autodoc]] GPT2ForQuestionAnswering\n    - forward\n\n## GPT2ForSequenceClassification\n\n[[autodoc]] GPT2ForSequenceClassification\n    - forward\n\n## GPT2ForTokenClassification\n\n[[autodoc]] GPT2ForTokenClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFGPT2Model\n\n[[autodoc]] TFGPT2Model\n    - call\n\n## TFGPT2LMHeadModel\n\n[[autodoc]] TFGPT2LMHeadModel\n    - call\n\n## TFGPT2DoubleHeadsModel\n\n[[autodoc]] TFGPT2DoubleHeadsModel\n    - call\n\n## TFGPT2ForSequenceClassification\n\n[[autodoc]] TFGPT2ForSequenceClassification\n    - call\n\n## TFSequenceClassifierOutputWithPast\n\n[[autodoc]] modeling_tf_outputs.TFSequenceClassifierOutputWithPast\n\n## TFGPT2Tokenizer\n\n[[autodoc]] TFGPT2Tokenizer\n\n</tf>\n<jax>\n\n## FlaxGPT2Model\n\n[[autodoc]] FlaxGPT2Model\n    - __call__\n\n## FlaxGPT2LMHeadModel\n\n[[autodoc]] FlaxGPT2LMHeadModel\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e8563fb7-54f2-4664-98e4-bab96d7546c0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e8563fb7-54f2-4664-98e4-bab96d7546c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef81e749-3739-47bf-8763-887fc68ff8ed\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4a58ff85-1052-4568-9fb8-29d7b50d8dad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"97d15610b46e52c22c5aef59a729fd3425ebfbce03423e9ce4f3efa9ef32b83f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3536, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "doc_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "ref_doc_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "chroma:document": "# BROS\n\n## Overview\n\nThe BROS model was proposed in [BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents](https://arxiv.org/abs/2108.04539) by Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, Sungrae Park.\n\nBROS stands for *BERT Relying On Spatiality*. It is an encoder-only Transformer model that takes a sequence of tokens and their bounding boxes as inputs and outputs a sequence of hidden states. BROS encode relative spatial information instead of using absolute spatial information.\n\nIt is pre-trained with two objectives: a token-masked language modeling objective (TMLM) used in BERT, and a novel area-masked language modeling objective (AMLM)\nIn TMLM, tokens are randomly masked, and the model predicts the masked tokens using spatial information and other unmasked tokens.\nAMLM is a 2D version of TMLM. It randomly masks text tokens and predicts with the same information as TMLM, but it masks text blocks (areas).\n\n`BrosForTokenClassification` has a simple linear layer on top of BrosModel. It predicts the label of each token.\n`BrosSpadeEEForTokenClassification` has an `initial_token_classifier` and `subsequent_token_classifier` on top of BrosModel. `initial_token_classifier` is used to predict the first token of each entity, and `subsequent_token_classifier` is used to predict the next token of within entity. `BrosSpadeELForTokenClassification` has an `entity_linker` on top of BrosModel. `entity_linker` is used to predict the relation between two entities.\n\n`BrosForTokenClassification` and `BrosSpadeEEForTokenClassification` essentially perform the same job. However, `BrosForTokenClassification` assumes input tokens are perfectly serialized (which is very challenging task since they exist in a 2D space), while `BrosSpadeEEForTokenClassification` allows for more flexibility in handling serialization errors as it predicts next connection tokens from one token.\n\n`BrosSpadeELForTokenClassification` perform the intra-entity linking task. It predicts relation from one token (of one entity) to another token (of another entity) if these two entities share some relation.\n\nBROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features.\n\nThe abstract from the paper is the following:\n\n*Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4a58ff85-1052-4568-9fb8-29d7b50d8dad
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4a58ff85-1052-4568-9fb8-29d7b50d8dad\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef81e749-3739-47bf-8763-887fc68ff8ed\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e8563fb7-54f2-4664-98e4-bab96d7546c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0b5d6965c6205045e767e448d585a8eb2337ebc11d893b94e68ced61dfcca0ba\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e01b30c7-b18d-455c-9e05-2d51bdda3b29\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a51d28b889aa12b15039a238fa0681ba8d697c6c02ef2efef96e03139551ee02\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1663, \"end_char_idx\": 4829, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "doc_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "ref_doc_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "chroma:document": "However, `BrosForTokenClassification` assumes input tokens are perfectly serialized (which is very challenging task since they exist in a 2D space), while `BrosSpadeEEForTokenClassification` allows for more flexibility in handling serialization errors as it predicts next connection tokens from one token.\n\n`BrosSpadeELForTokenClassification` perform the intra-entity linking task. It predicts relation from one token (of one entity) to another token (of another entity) if these two entities share some relation.\n\nBROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features.\n\nThe abstract from the paper is the following:\n\n*Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.*\n\nThis model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros).\n\n## Usage tips and examples\n\n- [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height.\n\n```python\ndef expand_and_normalize_bbox(bboxes, doc_width, doc_height):\n    # here, bboxes are numpy array\n\n    # Normalize bbox -> 0 ~ 1\n    bboxes[:, [0, 2]] = bboxes[:, [0, 2]] / width\n    bboxes[:, [1, 3]] = bboxes[:, [1, 3]] / height\n```\n\n- [`~transformers.BrosForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`] require not only `input_ids` and `bbox` but also `box_first_token_mask` for loss calculation. It is a mask to filter out non-first tokens of each box. You can obtain this mask by saving start token indices of bounding boxes when creating `input_ids` from words. You can make `box_first_token_mask` with following code,"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e01b30c7-b18d-455c-9e05-2d51bdda3b29
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e01b30c7-b18d-455c-9e05-2d51bdda3b29\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef81e749-3739-47bf-8763-887fc68ff8ed\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4a58ff85-1052-4568-9fb8-29d7b50d8dad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0a666a59f58a765d86aa516e27753bd154e4422ea8b740ffe0422bd69eea40be\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3298, \"end_char_idx\": 6382, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "doc_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "ref_doc_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "chroma:document": "This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.*\n\nThis model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros).\n\n## Usage tips and examples\n\n- [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height.\n\n```python\ndef expand_and_normalize_bbox(bboxes, doc_width, doc_height):\n    # here, bboxes are numpy array\n\n    # Normalize bbox -> 0 ~ 1\n    bboxes[:, [0, 2]] = bboxes[:, [0, 2]] / width\n    bboxes[:, [1, 3]] = bboxes[:, [1, 3]] / height\n```\n\n- [`~transformers.BrosForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`] require not only `input_ids` and `bbox` but also `box_first_token_mask` for loss calculation. It is a mask to filter out non-first tokens of each box. You can obtain this mask by saving start token indices of bounding boxes when creating `input_ids` from words. You can make `box_first_token_mask` with following code,\n\n\n```python\ndef make_box_first_token_mask(bboxes, words, tokenizer, max_seq_length=512):\n\n    box_first_token_mask = np.zeros(max_seq_length, dtype=np.bool_)\n\n    # encode(tokenize) each word from words (List[str])\n    input_ids_list: List[List[int]] = [tokenizer.encode(e, add_special_tokens=False) for e in words]\n\n    # get the length of each box\n    tokens_length_list: List[int] = [len(l) for l in input_ids_list]\n\n    box_end_token_indices = np.array(list(itertools.accumulate(tokens_length_list)))\n    box_start_token_indices = box_end_token_indices - np.array(tokens_length_list)\n\n    # filter out the indices that are out of max_seq_length\n    box_end_token_indices = box_end_token_indices[box_end_token_indices < max_seq_length - 1]\n    if len(box_start_token_indices) > len(box_end_token_indices):\n        box_start_token_indices = box_start_token_indices[: len(box_end_token_indices)]\n\n    # set box_start_token_indices to True\n    box_first_token_mask[box_start_token_indices] = True\n\n    return box_first_token_mask\n\n```\n\n## Resources\n\n- Demo scripts can be found [here](https://github.com/clovaai/bros).\n\n## BrosConfig\n\n[[autodoc]] BrosConfig\n\n## BrosProcessor\n\n[[autodoc]] BrosProcessor\n    - __call__\n\n## BrosModel\n\n[[autodoc]] BrosModel\n    - forward\n\n\n## BrosForTokenClassification\n\n[[autodoc]] BrosForTokenClassification\n    - forward\n\n## BrosSpadeEEForTokenClassification\n\n[[autodoc]] BrosSpadeEEForTokenClassification\n    - forward\n\n## BrosSpadeELForTokenClassification\n\n[[autodoc]] BrosSpadeELForTokenClassification\n    - forward"}


