# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e70d5322-d9ed-4e3a-b245-8c1c82decfc6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e70d5322-d9ed-4e3a-b245-8c1c82decfc6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"93cebb7f-4797-48ef-92a3-21f9fb96ebd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5d6f1937-bfa4-470e-ae37-a6c4d2c27d32\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"957ffcd5bc726e51a1e809dde0eeb28770d08eee0a151a02cede633d8adb39bc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f1a8b286-0d7a-4e7b-841e-e324e0b20c97\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"765584f10dd1fc2f119a4cc901eb46d8ec82eaa364f2d63e92e55595a2d1b5d1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4920, \"end_char_idx\": 7816, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "ref_doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "chroma:document": "### Batch Mixed Media Inference\n\nThe model can batch inputs composed of mixed samples of various types such as images, videos, and text. Here is an example.\n\n```python\n\nimage1 = Image.open(\"/path/to/image1.jpg\")\nimage2 = Image.open(\"/path/to/image2.jpg\")\nimage3 = Image.open(\"/path/to/image3.jpg\")\nimage4 = Image.open(\"/path/to/image4.jpg\")\nimage5 = Image.open(\"/path/to/image5.jpg\")\nvideo = fetch_video({\n    \"type\": \"video\",\n    \"video\": \"/path/to/video.mp4\",\n    \"fps\": 1.0\n})\n\n# Conversation for the first image\nconversation1 = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"Describe this image.\"}\n        ]\n    }\n]\n\n# Conversation with two images\nconversation2 = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is written in the pictures?\"}\n        ]\n    }\n]\n\n# Conversation with pure text\nconversation3 = [\n    {\n        \"role\": \"user\",\n        \"content\": \"who are you?\"\n    }\n]\n\n\n# Conversation with mixed midia\nconversation4 = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"image\"},\n            {\"type\": \"video\"},\n            {\"type\": \"text\", \"text\": \"What are the common elements in these medias?\"},\n        ],\n    }\n]\n\nconversations = [conversation1, conversation2, conversation3, conversation4]\n# Preparation for batch inference\ntexts = [processor.apply_chat_template(msg, add_generation_prompt=True) for msg in conversations]\ninputs = processor(\n    text=texts,\n    images=[image1, image2, image3, image4, image5],\n    videos=[video],\n    padding=True,\n    return_tensors=\"pt\",\n)\ninputs = inputs.to('cuda')\n\n# Batch Inference\noutput_ids = model.generate(**inputs, max_new_tokens=128)\ngenerated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]\noutput_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\nprint(output_text)\n```\n\n### Usage Tips\n\n#### Image Resolution for performance boost\n\nThe model supports a wide range of resolution inputs. By default, it uses the native resolution for input, but higher resolutions can enhance performance at the cost of more computation. Users can set the minimum and maximum number of pixels to achieve an optimal configuration for their needs.\n\n```python\n\nmin_pixels = 224*224\nmax_pixels = 2048*2048\nprocessor = AutoProcessor.from_pretrained(\"Qwen/Qwen2-VL-7B-Instruct\", min_pixels=min_pixels, max_pixels=max_pixels)\n\n```\n\n\n\n#### Multiple Image Inputs\n\nBy default, images and video content are directly included in the conversation. When handling multiple images, it's helpful to add labels to the images and videos for better reference. Users can control this behavior with the following settings:"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f1a8b286-0d7a-4e7b-841e-e324e0b20c97
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f1a8b286-0d7a-4e7b-841e-e324e0b20c97\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"93cebb7f-4797-48ef-92a3-21f9fb96ebd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e70d5322-d9ed-4e3a-b245-8c1c82decfc6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5d3acd7cad98a4b656a831c0237a9433bcf4e0a28c46ae1e4a11ed59b31ec64e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd2a7bef-e0e1-4235-aa1a-5e8adf2f675a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d36aa92fc769ee10e69019965ad0ec3d2d7600d37e14032eb8d02181e3fc6a26\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7543, \"end_char_idx\": 9653, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "ref_doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "chroma:document": "#### Multiple Image Inputs\n\nBy default, images and video content are directly included in the conversation. When handling multiple images, it's helpful to add labels to the images and videos for better reference. Users can control this behavior with the following settings:\n\n\n\n```python\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"}, \n            {\"type\": \"text\", \"text\": \"Hello, how are you?\"}\n        ]\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": \"I'm doing well, thank you for asking. How can I assist you today?\"\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Can you describe these images and video?\"}, \n            {\"type\": \"image\"}, \n            {\"type\": \"image\"}, \n            {\"type\": \"video\"}, \n            {\"type\": \"text\", \"text\": \"These are from my vacation.\"}\n        ]\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": \"I'd be happy to describe the images and video for you. Could you please provide more context about your vacation?\"\n    },\n    {\n        \"role\": \"user\",\n        \"content\": \"It was a trip to the mountains. Can you see the details in the images and video?\"\n    }\n]\n\n# default:\nprompt_without_id = processor.apply_chat_template(conversation, add_generation_prompt=True)\n# Excepted output: '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\n<|vision_start|><|image_pad|><|vision_end|>Hello, how are you?<|im_end|>\\n<|im_start|>assistant\\nI'm doing well, thank you for asking. How can I assist you today?<|im_end|>\\n<|im_start|>user\\nCan you describe these images and video?<|vision_start|><|image_pad|><|vision_end|><|vision_start|><|image_pad|><|vision_end|><|vision_start|><|video_pad|><|vision_end|>These are from my vacation.<|im_end|>\\n<|im_start|>assistant\\nI'd be happy to describe the images and video for you. Could you please provide more context about your vacation?<|im_end|>\\n<|im_start|>user\\nIt was a trip to the mountains. Can you see the details in the images and video?<|im_end|>\\n<|im_start|>assistant\\n'"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dd2a7bef-e0e1-4235-aa1a-5e8adf2f675a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dd2a7bef-e0e1-4235-aa1a-5e8adf2f675a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"93cebb7f-4797-48ef-92a3-21f9fb96ebd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f1a8b286-0d7a-4e7b-841e-e324e0b20c97\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"071fef8913cd1a27861d1ab971984659a9a07990a38c9a267b50737bc4bd06cf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9656, \"end_char_idx\": 11794, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "ref_doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "chroma:document": "# add ids\nprompt_with_id = processor.apply_chat_template(conversation, add_generation_prompt=True, add_vision_id=True)\n# Excepted output: '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\nPicture 1: <|vision_start|><|image_pad|><|vision_end|>Hello, how are you?<|im_end|>\\n<|im_start|>assistant\\nI'm doing well, thank you for asking. How can I assist you today?<|im_end|>\\n<|im_start|>user\\nCan you describe these images and video?Picture 2: <|vision_start|><|image_pad|><|vision_end|>Picture 3: <|vision_start|><|image_pad|><|vision_end|>Video 1: <|vision_start|><|video_pad|><|vision_end|>These are from my vacation.<|im_end|>\\n<|im_start|>assistant\\nI'd be happy to describe the images and video for you. Could you please provide more context about your vacation?<|im_end|>\\n<|im_start|>user\\nIt was a trip to the mountains. Can you see the details in the images and video?<|im_end|>\\n<|im_start|>assistant\\n'\n\n```\n\n#### Flash-Attention 2 to speed up generation\n\nFirst, make sure to install the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nAlso, you should have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.\n\nTo load and run a model using Flash Attention-2, simply add `attn_implementation=\"flash_attention_2\"` when loading the model as follows:\n\n```python\nfrom transformers import Qwen2VLForConditionalGeneration\n\nmodel = Qwen2VLForConditionalGeneration.from_pretrained(\n    \"Qwen/Qwen2-VL-7B-Instruct\", \n    torch_dtype=torch.bfloat16, \n    attn_implementation=\"flash_attention_2\",\n)\n```\n\n\n## Qwen2VLConfig\n\n[[autodoc]] Qwen2VLConfig\n\n## Qwen2VLImageProcessor\n\n[[autodoc]] Qwen2VLImageProcessor\n    - preprocess\n\n## Qwen2VLProcessor\n\n[[autodoc]] Qwen2VLProcessor\n\n## Qwen2VLModel\n\n[[autodoc]] Qwen2VLModel\n    - forward\n\n## Qwen2VLForConditionalGeneration\n\n[[autodoc]] Qwen2VLForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cd539eca-e2a4-4603-bde1-b2c6acb46e5a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cd539eca-e2a4-4603-bde1-b2c6acb46e5a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"70293822-19dc-4610-b50b-9f0c65b05d11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"55756f26bbf925ee6b445d381583fdc3b7afcc11cf98a7cb570bd3397581014e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"842e396a-f485-43bc-91d1-d0d43c66bfd1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6a9731a06952e7a5b0573498451d0fdf908e2a71fffc0d342a0e0be5c8ecf324\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2725, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "doc_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "ref_doc_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "chroma:document": "# SuperPoint\n\n## Overview\n\nThe SuperPoint model was proposed\nin [SuperPoint: Self-Supervised Interest Point Detection and Description](https://arxiv.org/abs/1712.07629) by Daniel\nDeTone, Tomasz Malisiewicz and Andrew Rabinovich.\n\nThis model is the result of a self-supervised training of a fully-convolutional network for interest point detection and\ndescription. The model is able to detect interest points that are repeatable under homographic transformations and\nprovide a descriptor for each point. The use of the model in its own is limited, but it can be used as a feature\nextractor for other tasks such as homography estimation, image matching, etc.\n\nThe abstract from the paper is the following:\n\n*This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a\nlarge number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our\nfully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and\nassociated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography\napproach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,\nsynthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able\nto repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other\ntraditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches\nwhen compared to LIFT, SIFT and ORB.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png\"\nalt=\"drawing\" width=\"500\"/>\n\n<small> SuperPoint overview. Taken from the <a href=\"https://arxiv.org/abs/1712.07629v4\">original paper.</a> </small>\n\n## Usage tips\n\nHere is a quick example of using the model to detect interest points in an image:\n\n```python\nfrom transformers import AutoImageProcessor, SuperPointForKeypointDetection\nimport torch\nfrom PIL import Image\nimport requests\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprocessor = AutoImageProcessor.from_pretrained(\"magic-leap-community/superpoint\")\nmodel = SuperPointForKeypointDetection.from_pretrained(\"magic-leap-community/superpoint\")\n\ninputs = processor(image, return_tensors=\"pt\")\noutputs = model(**inputs)\n```\n\nThe outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).\n\nYou can also feed multiple images to the model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 842e396a-f485-43bc-91d1-d0d43c66bfd1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"842e396a-f485-43bc-91d1-d0d43c66bfd1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"70293822-19dc-4610-b50b-9f0c65b05d11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"55756f26bbf925ee6b445d381583fdc3b7afcc11cf98a7cb570bd3397581014e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cd539eca-e2a4-4603-bde1-b2c6acb46e5a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4f6e5e9edf3cf5683a7d0d11efbc62f2bb2092fdb8c61eff3c9efcd337c49dc1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"44071b74-b6dd-4219-8101-3a709ad6a705\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"09da46a3b3df86abdc55d0ec6994a73466c32e30f9c1281aa502b712e3117a07\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1100, \"end_char_idx\": 4342, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "doc_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "ref_doc_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "chroma:document": "We introduce Homographic Adaptation, a multi-scale, multi-homography\napproach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,\nsynthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able\nto repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other\ntraditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches\nwhen compared to LIFT, SIFT and ORB.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png\"\nalt=\"drawing\" width=\"500\"/>\n\n<small> SuperPoint overview. Taken from the <a href=\"https://arxiv.org/abs/1712.07629v4\">original paper.</a> </small>\n\n## Usage tips\n\nHere is a quick example of using the model to detect interest points in an image:\n\n```python\nfrom transformers import AutoImageProcessor, SuperPointForKeypointDetection\nimport torch\nfrom PIL import Image\nimport requests\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprocessor = AutoImageProcessor.from_pretrained(\"magic-leap-community/superpoint\")\nmodel = SuperPointForKeypointDetection.from_pretrained(\"magic-leap-community/superpoint\")\n\ninputs = processor(image, return_tensors=\"pt\")\noutputs = model(**inputs)\n```\n\nThe outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).\n\nYou can also feed multiple images to the model. Due to the nature of SuperPoint, to output a dynamic number of keypoints,\nyou will need to use the mask attribute to retrieve the respective information :\n\n```python\nfrom transformers import AutoImageProcessor, SuperPointForKeypointDetection\nimport torch\nfrom PIL import Image\nimport requests\n\nurl_image_1 = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage_1 = Image.open(requests.get(url_image_1, stream=True).raw)\nurl_image_2 = \"http://images.cocodataset.org/test-stuff2017/000000000568.jpg\"\nimage_2 = Image.open(requests.get(url_image_2, stream=True).raw)\n\nimages = [image_1, image_2]\n\nprocessor = AutoImageProcessor.from_pretrained(\"magic-leap-community/superpoint\")\nmodel = SuperPointForKeypointDetection.from_pretrained(\"magic-leap-community/superpoint\")\n\ninputs = processor(images, return_tensors=\"pt\")\noutputs = model(**inputs)\n\nfor i in range(len(images)):\n    image_mask = outputs.mask[i]\n    image_indices = torch.nonzero(image_mask).squeeze()\n    image_keypoints = outputs.keypoints[i][image_indices]\n    image_scores = outputs.scores[i][image_indices]\n    image_descriptors = outputs.descriptors[i][image_indices]\n```\n\nYou can then print the keypoints on the image to visualize the result :\n```python\nimport cv2\nfor keypoint, score in zip(image_keypoints, image_scores):\n    keypoint_x, keypoint_y = int(keypoint[0].item()), int(keypoint[1].item())\n    color = tuple([score.item() * 255] * 3)\n    image = cv2.circle(image, (keypoint_x, keypoint_y), 2, color)\ncv2.imwrite(\"output_image.png\", image)\n```\n\nThis model was contributed by [stevenbucaille](https://huggingface.co/stevenbucaille)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 44071b74-b6dd-4219-8101-3a709ad6a705
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"44071b74-b6dd-4219-8101-3a709ad6a705\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"70293822-19dc-4610-b50b-9f0c65b05d11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"55756f26bbf925ee6b445d381583fdc3b7afcc11cf98a7cb570bd3397581014e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"842e396a-f485-43bc-91d1-d0d43c66bfd1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d1a4e4198be167c360e4faf1b0b0e392f2ba722562e0c9045f0fad1f12a4c36e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4343, \"end_char_idx\": 5256, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "doc_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "ref_doc_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "chroma:document": "The original code can be found [here](https://github.com/magicleap/SuperPointPretrainedNetwork).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SuperPoint. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A notebook showcasing inference and visualization with SuperPoint can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SuperPoint/Inference_with_SuperPoint_to_detect_interest_points_in_an_image.ipynb). \ud83c\udf0e\n\n## SuperPointConfig\n\n[[autodoc]] SuperPointConfig\n\n## SuperPointImageProcessor\n\n[[autodoc]] SuperPointImageProcessor\n\n- preprocess\n\n## SuperPointForKeypointDetection\n\n[[autodoc]] SuperPointForKeypointDetection\n\n- forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 44ef598c-79ba-42e1-9ccb-311bc92b3070
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flan-t5", "title": "FLAN-T5", "tokens": 610, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"44ef598c-79ba-42e1-9ccb-311bc92b3070\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flan-t5\", \"title\": \"FLAN-T5\", \"tokens\": 610, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"06c2897d-3181-42d2-ac47-bc64d060f004\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flan-t5\", \"title\": \"FLAN-T5\", \"tokens\": 610, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"052e45c889c784ad2c1f26dba252e6414d6b231f65ccd22d42dad8339a24f72b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1685, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "06c2897d-3181-42d2-ac47-bc64d060f004", "doc_id": "06c2897d-3181-42d2-ac47-bc64d060f004", "ref_doc_id": "06c2897d-3181-42d2-ac47-bc64d060f004", "chroma:document": "# FLAN-T5\n\n## Overview\n\nFLAN-T5 was released in the paper [Scaling Instruction-Finetuned Language Models](https://arxiv.org/pdf/2210.11416.pdf) - it is an enhanced version of T5 that has been finetuned in a mixture of tasks.\n\nOne can directly use FLAN-T5 weights without finetuning the model:\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/flan-t5-small\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/flan-t5-small\")\n\n>>> inputs = tokenizer(\"A step by step recipe to make bolognese pasta:\", return_tensors=\"pt\")\n>>> outputs = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['Pour a cup of bolognese into a large bowl and add the pasta']\n```\n\nFLAN-T5 includes the same improvements as T5 version 1.1 (see [here](https://huggingface.co/docs/transformers/model_doc/t5v1.1) for the full details of the model's improvements.)\n\nGoogle has released the following variants:\n\n- [google/flan-t5-small](https://huggingface.co/google/flan-t5-small)\n\n- [google/flan-t5-base](https://huggingface.co/google/flan-t5-base)\n\n- [google/flan-t5-large](https://huggingface.co/google/flan-t5-large)\n\n- [google/flan-t5-xl](https://huggingface.co/google/flan-t5-xl)\n\n- [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl).\n\nThe original checkpoints can be found [here](https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints).\n\n<Tip>\n\nRefer to [T5's documentation page](t5) for all API reference, code examples and notebooks. For more details regarding training and evaluation of the FLAN-T5, refer to the model card.\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b2f59fc8-8fd2-47d0-8289-0e21ec2b4d62
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b2f59fc8-8fd2-47d0-8289-0e21ec2b4d62\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"20533bdf-7520-43fb-9c1c-4ac90808f7d6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bcb5f3c5f15a803e49e6dc0dd3250147113bbf89e466bef5c5560eb852212c90\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aa9580dc-6f6b-466c-be79-e147d9dc94e5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"38303f5ac0f88881c98e64927f80261931afd54ef45b4dae38fd068a8640a888\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2864, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "20533bdf-7520-43fb-9c1c-4ac90808f7d6", "doc_id": "20533bdf-7520-43fb-9c1c-4ac90808f7d6", "ref_doc_id": "20533bdf-7520-43fb-9c1c-4ac90808f7d6", "chroma:document": "# CTRL\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=ctrl\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-ctrl-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/tiny-ctrl\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nCTRL model was proposed in [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858) by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and\nRichard Socher. It's a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus\nof ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.).\n\nThe abstract from the paper is the following:\n\n*Large-scale language models show promising text generation capabilities, but users cannot easily control particular\naspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model,\ntrained to condition on control codes that govern style, content, and task-specific behavior. Control codes were\nderived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while\nproviding more explicit control over text generation. These codes also allow CTRL to predict which parts of the\ntraining data are most likely given a sequence. This provides a potential method for analyzing large amounts of data\nvia model-based source attribution.*\n\nThis model was contributed by [keskarnitishr](https://huggingface.co/keskarnitishr). The original code can be found\n[here](https://github.com/salesforce/ctrl).\n\n## Usage tips\n\n- CTRL makes use of control codes to generate text: it requires generations to be started by certain words, sentences\n  or links to generate coherent text. Refer to the [original implementation](https://github.com/salesforce/ctrl) for\n  more information.\n- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- CTRL was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next\n  token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as it can be\n  observed in the *run_generation.py* example script.\n- The PyTorch models can take the `past_key_values` as input, which is the previously computed key/value attention pairs.\n  TensorFlow models accepts `past` as input. Using the `past_key_values` value prevents the model from re-computing\n  pre-computed values in the context of text generation. See the [`forward`](model_doc/ctrl#transformers.CTRLModel.forward)\n  method for more information on the usage of this argument."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: aa9580dc-6f6b-466c-be79-e147d9dc94e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"aa9580dc-6f6b-466c-be79-e147d9dc94e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"20533bdf-7520-43fb-9c1c-4ac90808f7d6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bcb5f3c5f15a803e49e6dc0dd3250147113bbf89e466bef5c5560eb852212c90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b2f59fc8-8fd2-47d0-8289-0e21ec2b4d62\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9f1879bacff36180959022e24a0a384264c8e5f42c182c18ed7e96183a7fa3ff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2867, \"end_char_idx\": 3600, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "20533bdf-7520-43fb-9c1c-4ac90808f7d6", "doc_id": "20533bdf-7520-43fb-9c1c-4ac90808f7d6", "ref_doc_id": "20533bdf-7520-43fb-9c1c-4ac90808f7d6", "chroma:document": "## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## CTRLConfig\n\n[[autodoc]] CTRLConfig\n\n## CTRLTokenizer\n\n[[autodoc]] CTRLTokenizer\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## CTRLModel\n\n[[autodoc]] CTRLModel\n    - forward\n\n## CTRLLMHeadModel\n\n[[autodoc]] CTRLLMHeadModel\n    - forward\n\n## CTRLForSequenceClassification\n\n[[autodoc]] CTRLForSequenceClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFCTRLModel\n\n[[autodoc]] TFCTRLModel\n    - call\n\n## TFCTRLLMHeadModel\n\n[[autodoc]] TFCTRLLMHeadModel\n    - call\n\n## TFCTRLForSequenceClassification\n\n[[autodoc]] TFCTRLForSequenceClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 46a364f3-b8ae-4c56-88a0-89cc519a63c9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/git", "title": "GIT", "tokens": 888, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"46a364f3-b8ae-4c56-88a0-89cc519a63c9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/git\", \"title\": \"GIT\", \"tokens\": 888, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"286c0ee5-7d2a-4e6b-8e41-6fed3bf0e2ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/git\", \"title\": \"GIT\", \"tokens\": 888, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8e695eda445ff55612b51c26331ca234eec8ab873d31219f8b7e7c098841f693\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3136, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "286c0ee5-7d2a-4e6b-8e41-6fed3bf0e2ee", "doc_id": "286c0ee5-7d2a-4e6b-8e41-6fed3bf0e2ee", "ref_doc_id": "286c0ee5-7d2a-4e6b-8e41-6fed3bf0e2ee", "chroma:document": "# GIT\n\n## Overview\n\nThe GIT model was proposed in [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by\nJianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. GIT is a decoder-only Transformer\nthat leverages [CLIP](clip)'s vision encoder to condition the model on vision inputs besides text. The model obtains state-of-the-art results on\nimage captioning and visual question answering benchmarks.\n\nThe abstract from the paper is the following:\n\n*In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/git_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> GIT architecture. Taken from the <a href=\"https://arxiv.org/abs/2205.14100\" target=\"_blank\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/GenerativeImage2Text).\n\n## Usage tips\n\n- GIT is implemented in a very similar way to GPT-2, the only difference being that the model is also conditioned on `pixel_values`.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GIT.\n\n- Demo notebooks regarding inference + fine-tuning GIT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GIT).\n- See also: [Causal language modeling task guide](../tasks/language_modeling)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## GitVisionConfig\n\n[[autodoc]] GitVisionConfig\n\n## GitVisionModel\n\n[[autodoc]] GitVisionModel\n    - forward\n\n## GitConfig\n\n[[autodoc]] GitConfig\n    - all\n\n## GitProcessor\n\n[[autodoc]] GitProcessor\n    - __call__\n\n## GitModel\n\n[[autodoc]] GitModel\n    - forward\n\n## GitForCausalLM\n\n[[autodoc]] GitForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9b28eec5-8a8e-4a50-a470-459971100f2c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clap", "title": "CLAP", "tokens": 857, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9b28eec5-8a8e-4a50-a470-459971100f2c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clap\", \"title\": \"CLAP\", \"tokens\": 857, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f4b4deb-bba4-42c1-9cd2-07f4bdd748fd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clap\", \"title\": \"CLAP\", \"tokens\": 857, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2952311c91ef6ca8ffcf841bfcec17524cfd1ce84e1e603f134c6e8d96ab6dce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3011, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f4b4deb-bba4-42c1-9cd2-07f4bdd748fd", "doc_id": "9f4b4deb-bba4-42c1-9cd2-07f4bdd748fd", "ref_doc_id": "9f4b4deb-bba4-42c1-9cd2-07f4bdd748fd", "chroma:document": "# CLAP\n\n## Overview\n\nThe CLAP model was proposed in [Large Scale Contrastive Language-Audio pretraining with\nfeature fusion and keyword-to-caption augmentation](https://arxiv.org/pdf/2211.06687.pdf) by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.\n\nCLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.\n\nThe abstract from the paper is the following:\n\n*Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-6*\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .\nThe original code can be found [here](https://github.com/LAION-AI/Clap).\n\n## ClapConfig\n\n[[autodoc]] ClapConfig\n    - from_text_audio_configs\n\n## ClapTextConfig\n\n[[autodoc]] ClapTextConfig\n\n## ClapAudioConfig\n\n[[autodoc]] ClapAudioConfig\n\n## ClapFeatureExtractor\n\n[[autodoc]] ClapFeatureExtractor\n\n## ClapProcessor\n\n[[autodoc]] ClapProcessor\n\n## ClapModel\n\n[[autodoc]] ClapModel\n    - forward\n    - get_text_features\n    - get_audio_features\n\n## ClapTextModel\n\n[[autodoc]] ClapTextModel\n    - forward\n\n## ClapTextModelWithProjection\n\n[[autodoc]] ClapTextModelWithProjection\n    - forward\n\n## ClapAudioModel\n\n[[autodoc]] ClapAudioModel\n    - forward\n\n## ClapAudioModelWithProjection\n\n[[autodoc]] ClapAudioModelWithProjection\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a90763d6-89e5-4dae-a594-d17b30591f2a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a90763d6-89e5-4dae-a594-d17b30591f2a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7189666f-ebfc-4324-bea3-f2880f59cee9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"90cdafd1166bebb76ce42f5770187f53677ad7409847c087ac3f35a265a53405\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"02e171af-0a96-4d54-83d9-bfc5c831dc58\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"10b1b79716168c7d477c667436e603efe4a3b6755e957eb60bc18aafe6014e1d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3218, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7189666f-ebfc-4324-bea3-f2880f59cee9", "doc_id": "7189666f-ebfc-4324-bea3-f2880f59cee9", "ref_doc_id": "7189666f-ebfc-4324-bea3-f2880f59cee9", "chroma:document": "# CLVP\n\n## Overview\n\nThe CLVP (Contrastive Language-Voice Pretrained Transformer) model was proposed in [Better speech synthesis through scaling](https://arxiv.org/abs/2305.07243) by James Betker.\n\nThe abstract from the paper is the following:\n\n*In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise - an expressive, multi-voice text-to-speech system.*\n\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\nThe original code can be found [here](https://github.com/neonbjb/tortoise-tts).\n\n\n## Usage tips\n\n1. CLVP is an integral part of the Tortoise TTS model.\n2. CLVP can be used to compare different generated speech candidates with the provided text, and the best speech tokens are forwarded to the diffusion model.\n3. The use of the [`ClvpModelForConditionalGeneration.generate()`] method is strongly recommended for tortoise usage.\n4. Note that the CLVP model expects the audio to be sampled at 22.05 kHz contrary to other audio models which expects 16 kHz. \n\n\n## Brief Explanation:\n\n- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.\n- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.\n- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.\n- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. \n- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. \n- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  \n\n\nExample :\n\n```python\n>>> import datasets\n>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration\n\n>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).\n>>> text = \"This is an example text.\"\n\n>>> ds = datasets.load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> ds = ds.cast_column(\"audio\", datasets.Audio(sampling_rate=22050))\n>>> sample = ds[0][\"audio\"]\n\n>>> # Define processor and model.\n>>> processor = ClvpProcessor.from_pretrained(\"susnato/clvp_dev\")\n>>> model = ClvpModelForConditionalGeneration.from_pretrained(\"susnato/clvp_dev\")\n\n>>> # Generate processor output and model output.\n>>> processor_output = processor(raw_speech=sample[\"array\"], sampling_rate=sample[\"sampling_rate\"], text=text, return_tensors=\"pt\")\n>>> generated_output = model.generate(**processor_output)\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 02e171af-0a96-4d54-83d9-bfc5c831dc58
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"02e171af-0a96-4d54-83d9-bfc5c831dc58\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7189666f-ebfc-4324-bea3-f2880f59cee9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"90cdafd1166bebb76ce42f5770187f53677ad7409847c087ac3f35a265a53405\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a90763d6-89e5-4dae-a594-d17b30591f2a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0cd7f60b27916b5d3501ae71dedfd75acc6c8d6d54d515de4a8db1901fdd91a7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1436, \"end_char_idx\": 3958, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7189666f-ebfc-4324-bea3-f2880f59cee9", "doc_id": "7189666f-ebfc-4324-bea3-f2880f59cee9", "ref_doc_id": "7189666f-ebfc-4324-bea3-f2880f59cee9", "chroma:document": "## Brief Explanation:\n\n- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.\n- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.\n- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.\n- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. \n- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. \n- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  \n\n\nExample :\n\n```python\n>>> import datasets\n>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration\n\n>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).\n>>> text = \"This is an example text.\"\n\n>>> ds = datasets.load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> ds = ds.cast_column(\"audio\", datasets.Audio(sampling_rate=22050))\n>>> sample = ds[0][\"audio\"]\n\n>>> # Define processor and model.\n>>> processor = ClvpProcessor.from_pretrained(\"susnato/clvp_dev\")\n>>> model = ClvpModelForConditionalGeneration.from_pretrained(\"susnato/clvp_dev\")\n\n>>> # Generate processor output and model output.\n>>> processor_output = processor(raw_speech=sample[\"array\"], sampling_rate=sample[\"sampling_rate\"], text=text, return_tensors=\"pt\")\n>>> generated_output = model.generate(**processor_output)\n```\n\n\n## ClvpConfig\n\n[[autodoc]] ClvpConfig\n    - from_sub_model_configs\n\n## ClvpEncoderConfig\n\n[[autodoc]] ClvpEncoderConfig\n\n## ClvpDecoderConfig\n\n[[autodoc]] ClvpDecoderConfig\n\n## ClvpTokenizer\n\n[[autodoc]] ClvpTokenizer\n    - save_vocabulary\n\n## ClvpFeatureExtractor\n\n[[autodoc]] ClvpFeatureExtractor\n    - __call__\n\n## ClvpProcessor\n\n[[autodoc]] ClvpProcessor\n    - __call__\n    - decode\n    - batch_decode\n\n## ClvpModelForConditionalGeneration\n\n[[autodoc]] ClvpModelForConditionalGeneration\n    - forward\n    - generate\n    - get_text_features\n    - get_speech_features\n\n## ClvpForCausalLM\n\n[[autodoc]] ClvpForCausalLM\n\n## ClvpModel\n\n[[autodoc]] ClvpModel\n\n## ClvpEncoder\n\n[[autodoc]] ClvpEncoder\n\n## ClvpDecoder\n\n[[autodoc]] ClvpDecoder"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0ef6d2b0-4a47-44a0-997b-b35701176cae
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/recurrent_gemma", "title": "RecurrentGemma", "tokens": 505, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0ef6d2b0-4a47-44a0-997b-b35701176cae\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/recurrent_gemma\", \"title\": \"RecurrentGemma\", \"tokens\": 505, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f3ea0dd8-a6fc-41d4-b197-e63e52f8a08e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/recurrent_gemma\", \"title\": \"RecurrentGemma\", \"tokens\": 505, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"baf5f71b6e8653a8aca416eef2e7a45d85b5901d8481ce2cadf41070e00d7585\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1507, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f3ea0dd8-a6fc-41d4-b197-e63e52f8a08e", "doc_id": "f3ea0dd8-a6fc-41d4-b197-e63e52f8a08e", "ref_doc_id": "f3ea0dd8-a6fc-41d4-b197-e63e52f8a08e", "chroma:document": "# RecurrentGemma\n\n## Overview\n\nThe Recurrent Gemma model was proposed in [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://storage.googleapis.com/deepmind-media/gemma/recurrentgemma-report.pdf) by the Griffin, RLHF and Gemma Teams of Google.\n\nThe abstract from the paper is the following:\n\n*We introduce RecurrentGemma, an open language model which uses Google\u2019s novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide a pre-trained model with 2B non-embedding parameters, and an instruction tuned variant. Both models achieve comparable performance to Gemma-2B despite being trained on fewer tokens.*\n\nTips:\n\n- The original checkpoints can be converted using the conversion script [`src/transformers/models/recurrent_gemma/convert_recurrent_gemma_weights_to_hf.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/recurrent_gemma/convert_recurrent_gemma_to_hf.py). \n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ). The original code can be found [here](https://github.com/google-deepmind/recurrentgemma).\n\n\n## RecurrentGemmaConfig\n\n[[autodoc]] RecurrentGemmaConfig\n\n\n## RecurrentGemmaModel\n\n[[autodoc]] RecurrentGemmaModel\n    - forward\n\n## RecurrentGemmaForCausalLM\n\n[[autodoc]] RecurrentGemmaForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bbb6d332-7e3d-47f4-9d9b-f0f2c7233293
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bbb6d332-7e3d-47f4-9d9b-f0f2c7233293\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08407bd5-9ae0-4945-8a2c-1223293ee091\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73a381bf-066c-4853-acb1-90fff736433f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7cc0bb5b80177fc43726039da8bbe670c7e2fc7261cb0c4965eaced23a0ba463\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3289, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "ref_doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "chroma:document": "# DETR\n\n## Overview\n\nThe DETR model was proposed in [End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872) by\nNicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. DETR\nconsists of a convolutional backbone followed by an encoder-decoder Transformer which can be trained end-to-end for\nobject detection. It greatly simplifies a lot of the complexity of models like Faster-R-CNN and Mask-R-CNN, which use\nthings like region proposals, non-maximum suppression procedure and anchor generation. Moreover, DETR can also be\nnaturally extended to perform panoptic segmentation, by simply adding a mask head on top of the decoder outputs.\n\nThe abstract from the paper is the following:\n\n*We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the\ndetection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression\nprocedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the\nnew framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via\nbipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries,\nDETR reasons about the relations of the objects and the global image context to directly output the final set of\npredictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many\nother modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and\nhighly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily\ngeneralized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive\nbaselines.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/facebookresearch/detr).\n\n## How DETR works\n\nHere's a TLDR explaining how [`~transformers.DetrForObjectDetection`] works:\n\nFirst, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use\nResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a\ntensor of shape `(batch_size, 3, height, width)`, assuming the image has 3 color channels (RGB). The CNN backbone\noutputs a new lower-resolution feature map, typically of shape `(batch_size, 2048, height/32, width/32)`. This is\nthen projected to match the hidden dimension of the Transformer of DETR, which is `256` by default, using a\n`nn.Conv2D` layer. So now, we have a tensor of shape `(batch_size, 256, height/32, width/32).` Next, the\nfeature map is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` =\n`(batch_size, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually\nlonger than usual, but with a smaller `d_model` (which in NLP is typically 768 or higher).\n\nNext, this is sent through the encoder, outputting `encoder_hidden_states` of the same shape (you can consider\nthese as image features)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 73a381bf-066c-4853-acb1-90fff736433f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"73a381bf-066c-4853-acb1-90fff736433f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08407bd5-9ae0-4945-8a2c-1223293ee091\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bbb6d332-7e3d-47f4-9d9b-f0f2c7233293\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8133511a188276b66d7ff74afe79471446075eaf74533ef3921005f4f451f77c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f6c9dbb1-d626-4305-aecd-7258e302de88\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"23941821e25ce2b1aa060ae85a540c4ab9aca2358db261fdbc09e5e92424a9fd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1778, \"end_char_idx\": 5025, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "ref_doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "chroma:document": "Moreover, DETR can be easily\ngeneralized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive\nbaselines.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/facebookresearch/detr).\n\n## How DETR works\n\nHere's a TLDR explaining how [`~transformers.DetrForObjectDetection`] works:\n\nFirst, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use\nResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a\ntensor of shape `(batch_size, 3, height, width)`, assuming the image has 3 color channels (RGB). The CNN backbone\noutputs a new lower-resolution feature map, typically of shape `(batch_size, 2048, height/32, width/32)`. This is\nthen projected to match the hidden dimension of the Transformer of DETR, which is `256` by default, using a\n`nn.Conv2D` layer. So now, we have a tensor of shape `(batch_size, 256, height/32, width/32).` Next, the\nfeature map is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` =\n`(batch_size, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually\nlonger than usual, but with a smaller `d_model` (which in NLP is typically 768 or higher).\n\nNext, this is sent through the encoder, outputting `encoder_hidden_states` of the same shape (you can consider\nthese as image features). Next, so-called **object queries** are sent through the decoder. This is a tensor of shape\n`(batch_size, num_queries, d_model)`, with `num_queries` typically set to 100 and initialized with zeros.\nThese input embeddings are learnt positional encodings that the authors refer to as object queries, and similarly to\nthe encoder, they are added to the input of each attention layer. Each object query will look for a particular object\nin the image. The decoder updates these embeddings through multiple self-attention and encoder-decoder attention layers\nto output `decoder_hidden_states` of the same shape: `(batch_size, num_queries, d_model)`. Next, two heads\nare added on top for object detection: a linear layer for classifying each object query into one of the objects or \"no\nobject\", and a MLP to predict bounding boxes for each query.\n\nThe model is trained using a **bipartite matching loss**: so what we actually do is compare the predicted classes +\nbounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N\n(so if an image only contains 4 objects, 96 annotations will just have a \"no object\" as class and \"no bounding box\" as\nbounding box). The [Hungarian matching algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) is used to find\nan optimal one-to-one mapping of each of the N queries to each of the N annotations. Next, standard cross-entropy (for\nthe classes) and a linear combination of the L1 and [generalized IoU loss](https://giou.stanford.edu/) (for the\nbounding boxes) are used to optimize the parameters of the model.\n\nDETR can be naturally extended to perform panoptic segmentation (which unifies semantic segmentation and instance\nsegmentation)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f6c9dbb1-d626-4305-aecd-7258e302de88
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f6c9dbb1-d626-4305-aecd-7258e302de88\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08407bd5-9ae0-4945-8a2c-1223293ee091\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73a381bf-066c-4853-acb1-90fff736433f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d88b6fe2d8433ef1c414f3be72a0966e68d29a0d4b63b761530a8641d64b65cf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"14e7640e-f037-4acd-8345-c0021c5b369a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"981778274d11ee1aaa6676a47f03930acc7a7e47bf9b12c3828c00c246a9aa6f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3290, \"end_char_idx\": 6686, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "ref_doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "chroma:document": "Next, so-called **object queries** are sent through the decoder. This is a tensor of shape\n`(batch_size, num_queries, d_model)`, with `num_queries` typically set to 100 and initialized with zeros.\nThese input embeddings are learnt positional encodings that the authors refer to as object queries, and similarly to\nthe encoder, they are added to the input of each attention layer. Each object query will look for a particular object\nin the image. The decoder updates these embeddings through multiple self-attention and encoder-decoder attention layers\nto output `decoder_hidden_states` of the same shape: `(batch_size, num_queries, d_model)`. Next, two heads\nare added on top for object detection: a linear layer for classifying each object query into one of the objects or \"no\nobject\", and a MLP to predict bounding boxes for each query.\n\nThe model is trained using a **bipartite matching loss**: so what we actually do is compare the predicted classes +\nbounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N\n(so if an image only contains 4 objects, 96 annotations will just have a \"no object\" as class and \"no bounding box\" as\nbounding box). The [Hungarian matching algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) is used to find\nan optimal one-to-one mapping of each of the N queries to each of the N annotations. Next, standard cross-entropy (for\nthe classes) and a linear combination of the L1 and [generalized IoU loss](https://giou.stanford.edu/) (for the\nbounding boxes) are used to optimize the parameters of the model.\n\nDETR can be naturally extended to perform panoptic segmentation (which unifies semantic segmentation and instance\nsegmentation). [`~transformers.DetrForSegmentation`] adds a segmentation mask head on top of\n[`~transformers.DetrForObjectDetection`]. The mask head can be trained either jointly, or in a two steps process,\nwhere one first trains a [`~transformers.DetrForObjectDetection`] model to detect bounding boxes around both\n\"things\" (instances) and \"stuff\" (background things like trees, roads, sky), then freeze all the weights and train only\nthe mask head for 25 epochs. Experimentally, these two approaches give similar results. Note that predicting boxes is\nrequired for the training to be possible, since the Hungarian matching is computed using distances between boxes.\n\n## Usage tips\n\n- DETR uses so-called **object queries** to detect objects in an image. The number of queries determines the maximum\n  number of objects that can be detected in a single image, and is set to 100 by default (see parameter\n  `num_queries` of [`~transformers.DetrConfig`]). Note that it's good to have some slack (in COCO, the\n  authors used 100, while the maximum number of objects in a COCO image is ~70).\n- The decoder of DETR updates the query embeddings in parallel. This is different from language models like GPT-2,\n  which use autoregressive decoding instead of parallel. Hence, no causal attention mask is used.\n- DETR adds position embeddings to the hidden states at each self-attention and cross-attention layer before projecting\n  to queries and keys. For the position embeddings of the image, one can choose between fixed sinusoidal or learned\n  absolute position embeddings. By default, the parameter `position_embedding_type` of\n  [`~transformers.DetrConfig`] is set to `\"sine\"`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 14e7640e-f037-4acd-8345-c0021c5b369a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"14e7640e-f037-4acd-8345-c0021c5b369a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08407bd5-9ae0-4945-8a2c-1223293ee091\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6c9dbb1-d626-4305-aecd-7258e302de88\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9a4bdd12fe0a50f218bc637639c1fc6c845308868ac1bf2bf8c9fc83828da0d6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"90e3a577-58e4-494d-9255-fc3a7e8e30cd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e8abbb78a57232433b5617e627f1746cdae35cb8de3362664cd2bfdd92f3f0bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5026, \"end_char_idx\": 8320, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "ref_doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "chroma:document": "[`~transformers.DetrForSegmentation`] adds a segmentation mask head on top of\n[`~transformers.DetrForObjectDetection`]. The mask head can be trained either jointly, or in a two steps process,\nwhere one first trains a [`~transformers.DetrForObjectDetection`] model to detect bounding boxes around both\n\"things\" (instances) and \"stuff\" (background things like trees, roads, sky), then freeze all the weights and train only\nthe mask head for 25 epochs. Experimentally, these two approaches give similar results. Note that predicting boxes is\nrequired for the training to be possible, since the Hungarian matching is computed using distances between boxes.\n\n## Usage tips\n\n- DETR uses so-called **object queries** to detect objects in an image. The number of queries determines the maximum\n  number of objects that can be detected in a single image, and is set to 100 by default (see parameter\n  `num_queries` of [`~transformers.DetrConfig`]). Note that it's good to have some slack (in COCO, the\n  authors used 100, while the maximum number of objects in a COCO image is ~70).\n- The decoder of DETR updates the query embeddings in parallel. This is different from language models like GPT-2,\n  which use autoregressive decoding instead of parallel. Hence, no causal attention mask is used.\n- DETR adds position embeddings to the hidden states at each self-attention and cross-attention layer before projecting\n  to queries and keys. For the position embeddings of the image, one can choose between fixed sinusoidal or learned\n  absolute position embeddings. By default, the parameter `position_embedding_type` of\n  [`~transformers.DetrConfig`] is set to `\"sine\"`.\n- During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help\n  the model output the correct number of objects of each class. If you set the parameter `auxiliary_loss` of\n  [`~transformers.DetrConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses\n  are added after each decoder layer (with the FFNs sharing parameters).\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  _num_boxes_ variable in the _DetrLoss_ class of _modeling_detr.py_. When training on multiple nodes, this should be\n  set to the average number of target boxes across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/models/detr.py#L227-L232).\n- [`~transformers.DetrForObjectDetection`] and [`~transformers.DetrForSegmentation`] can be initialized with\n  any convolutional backbone available in the [timm library](https://github.com/rwightman/pytorch-image-models).\n  Initializing with a MobileNet backbone for example can be done by setting the `backbone` attribute of\n  [`~transformers.DetrConfig`] to `\"tf_mobilenetv3_small_075\"`, and then initializing the model with that\n  config.\n- DETR resizes the input images such that the shortest side is at least a certain amount of pixels while the longest is\n  at most 1333 pixels. At training time, scale augmentation is used such that the shortest side is randomly set to at\n  least 480 and at most 800 pixels. At inference time, the shortest side is set to 800."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 90e3a577-58e4-494d-9255-fc3a7e8e30cd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"90e3a577-58e4-494d-9255-fc3a7e8e30cd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08407bd5-9ae0-4945-8a2c-1223293ee091\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"14e7640e-f037-4acd-8345-c0021c5b369a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"dbbc7498d8e892fbd60aeb558160e59efd14e2771004e528688d6e0f1682e134\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50ff22a9-9db6-45bd-90e2-3f6e5c9eaae3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6659784809a4f14b87e33413e6ae2022780aaaae3758eabb9f9574ab3e1e0b60\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6869, \"end_char_idx\": 10172, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "ref_doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "chroma:document": "If you set the parameter `auxiliary_loss` of\n  [`~transformers.DetrConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses\n  are added after each decoder layer (with the FFNs sharing parameters).\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  _num_boxes_ variable in the _DetrLoss_ class of _modeling_detr.py_. When training on multiple nodes, this should be\n  set to the average number of target boxes across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/models/detr.py#L227-L232).\n- [`~transformers.DetrForObjectDetection`] and [`~transformers.DetrForSegmentation`] can be initialized with\n  any convolutional backbone available in the [timm library](https://github.com/rwightman/pytorch-image-models).\n  Initializing with a MobileNet backbone for example can be done by setting the `backbone` attribute of\n  [`~transformers.DetrConfig`] to `\"tf_mobilenetv3_small_075\"`, and then initializing the model with that\n  config.\n- DETR resizes the input images such that the shortest side is at least a certain amount of pixels while the longest is\n  at most 1333 pixels. At training time, scale augmentation is used such that the shortest side is randomly set to at\n  least 480 and at most 800 pixels. At inference time, the shortest side is set to 800. One can use\n  [`~transformers.DetrImageProcessor`] to prepare images (and optional annotations in COCO format) for the\n  model. Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the\n  largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.\n  Alternatively, one can also define a custom `collate_fn` in order to batch images together, using\n  [`~transformers.DetrImageProcessor.pad_and_create_pixel_mask`].\n- The size of the images will determine the amount of memory being used, and will thus determine the `batch_size`.\n  It is advised to use a batch size of 2 per GPU. See [this Github thread](https://github.com/facebookresearch/detr/issues/150) for more info.\n\nThere are three ways to instantiate a DETR model (depending on what you prefer):\n\nOption 1: Instantiate DETR with pre-trained weights for entire model\n```py\n>>> from transformers import DetrForObjectDetection\n\n>>> model = DetrForObjectDetection.from_pretrained(\"facebook/detr-resnet-50\")\n```\n\nOption 2: Instantiate DETR with randomly initialized weights for Transformer, but pre-trained weights for backbone\n```py\n>>> from transformers import DetrConfig, DetrForObjectDetection\n\n>>> config = DetrConfig()\n>>> model = DetrForObjectDetection(config)\n```\nOption 3: Instantiate DETR with randomly initialized weights for backbone + Transformer\n```py\n>>> config = DetrConfig(use_pretrained_backbone=False)\n>>> model = DetrForObjectDetection(config)\n```\n\nAs a summary, consider the following table:\n\n| Task | Object detection | Instance segmentation | Panoptic segmentation |\n|------|------------------|-----------------------|-----------------------|\n| **Description** | Predicting bounding boxes and class labels around objects in an image | Predicting masks around objects (i.e."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 50ff22a9-9db6-45bd-90e2-3f6e5c9eaae3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50ff22a9-9db6-45bd-90e2-3f6e5c9eaae3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08407bd5-9ae0-4945-8a2c-1223293ee091\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"90e3a577-58e4-494d-9255-fc3a7e8e30cd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9ae0219487690477498644579e5a3168eea572fd4b45b24f15445e6c7eb37cdb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1d8e564f-0c72-411d-8795-0b04d5ec63f6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c095f323596f810559cad014578152e749735cd92decc98e9f4cd0d7b7291f54\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8449, \"end_char_idx\": 11155, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "ref_doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "chroma:document": "Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the\n  largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.\n  Alternatively, one can also define a custom `collate_fn` in order to batch images together, using\n  [`~transformers.DetrImageProcessor.pad_and_create_pixel_mask`].\n- The size of the images will determine the amount of memory being used, and will thus determine the `batch_size`.\n  It is advised to use a batch size of 2 per GPU. See [this Github thread](https://github.com/facebookresearch/detr/issues/150) for more info.\n\nThere are three ways to instantiate a DETR model (depending on what you prefer):\n\nOption 1: Instantiate DETR with pre-trained weights for entire model\n```py\n>>> from transformers import DetrForObjectDetection\n\n>>> model = DetrForObjectDetection.from_pretrained(\"facebook/detr-resnet-50\")\n```\n\nOption 2: Instantiate DETR with randomly initialized weights for Transformer, but pre-trained weights for backbone\n```py\n>>> from transformers import DetrConfig, DetrForObjectDetection\n\n>>> config = DetrConfig()\n>>> model = DetrForObjectDetection(config)\n```\nOption 3: Instantiate DETR with randomly initialized weights for backbone + Transformer\n```py\n>>> config = DetrConfig(use_pretrained_backbone=False)\n>>> model = DetrForObjectDetection(config)\n```\n\nAs a summary, consider the following table:\n\n| Task | Object detection | Instance segmentation | Panoptic segmentation |\n|------|------------------|-----------------------|-----------------------|\n| **Description** | Predicting bounding boxes and class labels around objects in an image | Predicting masks around objects (i.e. instances) in an image | Predicting masks around both objects (i.e. instances) as well as \"stuff\" (i.e. background things like trees and roads) in an image |\n| **Model** | [`~transformers.DetrForObjectDetection`] | [`~transformers.DetrForSegmentation`] | [`~transformers.DetrForSegmentation`] |\n| **Example dataset** | COCO detection | COCO detection, COCO panoptic | COCO panoptic  |                                                                        |\n| **Format of annotations to provide to**  [`~transformers.DetrImageProcessor`] | {'image_id': `int`, 'annotations': `List[Dict]`} each Dict being a COCO object annotation  | {'image_id': `int`, 'annotations': `List[Dict]`}  (in case of COCO detection) or {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} (in case of COCO panoptic) | {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} and masks_path (path to directory containing PNG files of the masks) |\n| **Postprocessing** (i.e."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1d8e564f-0c72-411d-8795-0b04d5ec63f6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1d8e564f-0c72-411d-8795-0b04d5ec63f6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08407bd5-9ae0-4945-8a2c-1223293ee091\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50ff22a9-9db6-45bd-90e2-3f6e5c9eaae3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"21c43483412201cf1ef457d256d1ab7ab66cdd8e06071d2da8fda6e4c6fec1a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"37b1ae69-2a82-4a99-a7a3-9eabdc34d35f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"160e1d9ce59d6d183ea4fafeaca9384dca7efdad9bec24cfb6e4c439882240ac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10173, \"end_char_idx\": 13285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "ref_doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "chroma:document": "instances) in an image | Predicting masks around both objects (i.e. instances) as well as \"stuff\" (i.e. background things like trees and roads) in an image |\n| **Model** | [`~transformers.DetrForObjectDetection`] | [`~transformers.DetrForSegmentation`] | [`~transformers.DetrForSegmentation`] |\n| **Example dataset** | COCO detection | COCO detection, COCO panoptic | COCO panoptic  |                                                                        |\n| **Format of annotations to provide to**  [`~transformers.DetrImageProcessor`] | {'image_id': `int`, 'annotations': `List[Dict]`} each Dict being a COCO object annotation  | {'image_id': `int`, 'annotations': `List[Dict]`}  (in case of COCO detection) or {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} (in case of COCO panoptic) | {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} and masks_path (path to directory containing PNG files of the masks) |\n| **Postprocessing** (i.e. converting the output of the model to Pascal VOC format) | [`~transformers.DetrImageProcessor.post_process`] | [`~transformers.DetrImageProcessor.post_process_segmentation`] | [`~transformers.DetrImageProcessor.post_process_segmentation`], [`~transformers.DetrImageProcessor.post_process_panoptic`] |\n| **evaluators** | `CocoEvaluator` with `iou_types=\"bbox\"` | `CocoEvaluator` with `iou_types=\"bbox\"` or `\"segm\"` | `CocoEvaluator` with `iou_tupes=\"bbox\"` or `\"segm\"`, `PanopticEvaluator` |\n\nIn short, one should prepare the data either in COCO detection or COCO panoptic format, then use\n[`~transformers.DetrImageProcessor`] to create `pixel_values`, `pixel_mask` and optional\n`labels`, which can then be used to train (or fine-tune) a model. For evaluation, one should first convert the\noutputs of the model using one of the postprocessing methods of [`~transformers.DetrImageProcessor`]. These can\nbe be provided to either `CocoEvaluator` or `PanopticEvaluator`, which allow you to calculate metrics like\nmean Average Precision (mAP) and Panoptic Quality (PQ). The latter objects are implemented in the [original repository](https://github.com/facebookresearch/detr). See the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR) for more info regarding evaluation.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DETR.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- All example notebooks illustrating fine-tuning [`DetrForObjectDetection`] and [`DetrForSegmentation`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR).\n- Scripts for finetuning [`DetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 37b1ae69-2a82-4a99-a7a3-9eabdc34d35f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"37b1ae69-2a82-4a99-a7a3-9eabdc34d35f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08407bd5-9ae0-4945-8a2c-1223293ee091\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1d8e564f-0c72-411d-8795-0b04d5ec63f6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9c66f17adb24b4adf81c47bc48208328afb22545b7a85d9b0eb9811e1cc8b002\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11900, \"end_char_idx\": 14294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "ref_doc_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "chroma:document": "For evaluation, one should first convert the\noutputs of the model using one of the postprocessing methods of [`~transformers.DetrImageProcessor`]. These can\nbe be provided to either `CocoEvaluator` or `PanopticEvaluator`, which allow you to calculate metrics like\nmean Average Precision (mAP) and Panoptic Quality (PQ). The latter objects are implemented in the [original repository](https://github.com/facebookresearch/detr). See the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR) for more info regarding evaluation.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DETR.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- All example notebooks illustrating fine-tuning [`DetrForObjectDetection`] and [`DetrForSegmentation`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR).\n- Scripts for finetuning [`DetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DetrConfig\n\n[[autodoc]] DetrConfig\n\n## DetrImageProcessor\n\n[[autodoc]] DetrImageProcessor\n    - preprocess\n    - post_process_object_detection\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## DetrFeatureExtractor\n\n[[autodoc]] DetrFeatureExtractor\n    - __call__\n    - post_process_object_detection\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## DETR specific outputs\n\n[[autodoc]] models.detr.modeling_detr.DetrModelOutput\n\n[[autodoc]] models.detr.modeling_detr.DetrObjectDetectionOutput\n\n[[autodoc]] models.detr.modeling_detr.DetrSegmentationOutput\n\n## DetrModel\n\n[[autodoc]] DetrModel\n    - forward\n\n## DetrForObjectDetection\n\n[[autodoc]] DetrForObjectDetection\n    - forward\n\n## DetrForSegmentation\n\n[[autodoc]] DetrForSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e3d6cb1a-48ce-4345-a881-9471b9d62f69
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlnet", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e3d6cb1a-48ce-4345-a881-9471b9d62f69\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ce2f9fba-be50-433d-af58-688a84d9f5d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3b1fc93b1f1103c5494c134c9e10f5c2ef842dd66b1f399feac95d087cc4acbb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"769dedf0-ca19-4d35-8020-0804f2bf5616\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6852a40e77a730b72069745d1c2dca914293210f9552d58125fa06d09af8b6de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3283, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ce2f9fba-be50-433d-af58-688a84d9f5d7", "doc_id": "ce2f9fba-be50-433d-af58-688a84d9f5d7", "ref_doc_id": "ce2f9fba-be50-433d-af58-688a84d9f5d7", "chroma:document": "# XLNet\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=xlnet\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-xlnet-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/xlnet-base-cased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe XLNet model was proposed in [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov,\nQuoc V. Le. XLnet is an extension of the Transformer-XL model pre-trained using an autoregressive method to learn\nbidirectional contexts by maximizing the expected likelihood over all permutations of the input sequence factorization\norder.\n\nThe abstract from the paper is the following:\n\n*With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves\nbetter performance than pretraining approaches based on autoregressive language modeling. However, relying on\ncorrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a\npretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive\npretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all\npermutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive\nformulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into\npretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large\nmargin, including question answering, natural language inference, sentiment analysis, and document ranking.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/zihangdai/xlnet/).\n\n## Usage tips\n\n- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.\n- Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained\n  using only a sub-set of the output tokens as target which are selected with the `target_mapping` input.\n- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and\n  `target_mapping` inputs to control the attention span and outputs (see examples in\n  *examples/pytorch/text-generation/run_generation.py*)\n- XLNet is one of the few models that has no sequence length limit.\n- XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done with a mask, the sentence is actually fed in the model in the right order, but instead of masking the first n tokens for n+1, XLNet uses a mask that hides the previous tokens in some given permutation of 1,\u2026,sequence length.\n- XLNet also uses the same recurrence mechanism as Transformer-XL to build long-term dependencies."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 769dedf0-ca19-4d35-8020-0804f2bf5616
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlnet", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"769dedf0-ca19-4d35-8020-0804f2bf5616\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ce2f9fba-be50-433d-af58-688a84d9f5d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3b1fc93b1f1103c5494c134c9e10f5c2ef842dd66b1f399feac95d087cc4acbb\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e3d6cb1a-48ce-4345-a881-9471b9d62f69\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"22a111c93804259ceffc1764e3861399b9f609af9d8118e9b4e6bec61bd7b6f4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1577, \"end_char_idx\": 5969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ce2f9fba-be50-433d-af58-688a84d9f5d7", "doc_id": "ce2f9fba-be50-433d-af58-688a84d9f5d7", "ref_doc_id": "ce2f9fba-be50-433d-af58-688a84d9f5d7", "chroma:document": "Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into\npretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large\nmargin, including question answering, natural language inference, sentiment analysis, and document ranking.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/zihangdai/xlnet/).\n\n## Usage tips\n\n- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.\n- Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained\n  using only a sub-set of the output tokens as target which are selected with the `target_mapping` input.\n- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and\n  `target_mapping` inputs to control the attention span and outputs (see examples in\n  *examples/pytorch/text-generation/run_generation.py*)\n- XLNet is one of the few models that has no sequence length limit.\n- XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done with a mask, the sentence is actually fed in the model in the right order, but instead of masking the first n tokens for n+1, XLNet uses a mask that hides the previous tokens in some given permutation of 1,\u2026,sequence length.\n- XLNet also uses the same recurrence mechanism as Transformer-XL to build long-term dependencies.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## XLNetConfig\n\n[[autodoc]] XLNetConfig\n\n## XLNetTokenizer\n\n[[autodoc]] XLNetTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## XLNetTokenizerFast\n\n[[autodoc]] XLNetTokenizerFast\n\n## XLNet specific outputs\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetModelOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetLMHeadModelOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForSequenceClassificationOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForMultipleChoiceOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForTokenClassificationOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringSimpleOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetModelOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetLMHeadModelOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForSequenceClassificationOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForMultipleChoiceOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForTokenClassificationOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForQuestionAnsweringSimpleOutput\n\n<frameworkcontent>\n<pt>\n\n## XLNetModel\n\n[[autodoc]] XLNetModel\n    - forward\n\n## XLNetLMHeadModel\n\n[[autodoc]] XLNetLMHeadModel\n    - forward\n\n## XLNetForSequenceClassification\n\n[[autodoc]] XLNetForSequenceClassification\n    - forward\n\n## XLNetForMultipleChoice\n\n[[autodoc]] XLNetForMultipleChoice\n    - forward\n\n## XLNetForTokenClassification\n\n[[autodoc]] XLNetForTokenClassification\n    - forward\n\n## XLNetForQuestionAnsweringSimple\n\n[[autodoc]] XLNetForQuestionAnsweringSimple\n    - forward\n\n## XLNetForQuestionAnswering\n\n[[autodoc]] XLNetForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFXLNetModel\n\n[[autodoc]] TFXLNetModel\n    - call\n\n## TFXLNetLMHeadModel\n\n[[autodoc]] TFXLNetLMHeadModel\n    - call\n\n## TFXLNetForSequenceClassification\n\n[[autodoc]] TFXLNetForSequenceClassification\n    - call\n\n## TFLNetForMultipleChoice\n\n[[autodoc]] TFXLNetForMultipleChoice\n    - call\n\n## TFXLNetForTokenClassification\n\n[[autodoc]] TFXLNetForTokenClassification\n    - call\n\n## TFXLNetForQuestionAnsweringSimple\n\n[[autodoc]] TFXLNetForQuestionAnsweringSimple\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9501de56-f184-47ef-bc3f-726e45a32dcf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9501de56-f184-47ef-bc3f-726e45a32dcf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0e3b34c-6620-449f-b66a-4ff56ae82c6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d7db83f3663ada04d53c9f6c94dbd5fc26f849ba45c8d94d4da96501d5ad7087\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"14f61be5-9701-4ddd-8c26-df7770335bdc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0b564a1142000f05d18fb6d5c78e5f38ee094ba146a5af9cefc2af2d15d4a299\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3707, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "doc_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "ref_doc_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "chroma:document": "# ESM\n\n## Overview\n\nThis page provides code and pre-trained weights for Transformer protein language models from Meta AI's Fundamental \nAI Research Team, providing the state-of-the-art ESMFold and ESM-2, and the previously released ESM-1b and ESM-1v.\nTransformer protein language models were introduced in the paper [Biological structure and function emerge from scaling\nunsupervised learning to 250 million protein sequences](https://www.pnas.org/content/118/15/e2016239118) by \nAlexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, \nC. Lawrence Zitnick, Jerry Ma, and Rob Fergus.\nThe first version of this paper was [preprinted in 2019](https://www.biorxiv.org/content/10.1101/622803v1?versioned=true).\n\nESM-2 outperforms all tested single-sequence protein language models across a range of structure prediction tasks,\nand enables atomic resolution structure prediction.\nIt was released with the paper [Language models of protein sequences at the scale of evolution enable accurate\nstructure prediction](https://doi.org/10.1101/2022.07.20.500902) by Zeming Lin, Halil Akin, Roshan Rao, Brian Hie,\nZhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido and Alexander Rives.\n\nAlso introduced in this paper was ESMFold. It uses an ESM-2 stem with a head that can predict folded protein\nstructures with state-of-the-art accuracy. Unlike [AlphaFold2](https://www.nature.com/articles/s41586-021-03819-2),\nit relies on the token embeddings from the large pre-trained protein language model stem and does not perform a multiple\nsequence alignment (MSA) step at inference time, which means that ESMFold checkpoints are fully \"standalone\" -\nthey do not require a database of known protein sequences and structures with associated external query tools\nto make predictions, and are much faster as a result.\n\n\nThe abstract from \n\"Biological structure and function emerge from scaling unsupervised learning to 250 \nmillion protein sequences\" is\n\n\n*In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised\nlearning has led to major advances in representation learning and statistical generation. In the life sciences, the\nanticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling\nat the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To\nthis end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250\nmillion protein sequences spanning evolutionary diversity. The resulting model contains information about biological\nproperties in its representations. The representations are learned from sequence data alone. The learned representation\nspace has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to\nremote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and\ncan be identified by linear projections. Representation learning produces features that generalize across a range of\napplications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and\nimproving state-of-the-art features for long-range contact prediction.*\n\n\nThe abstract from\n\"Language models of protein sequences at the scale of evolution enable accurate structure prediction\" is\n\n*Large language models have recently been shown to develop emergent capabilities with scale, going beyond\nsimple pattern matching to perform higher level reasoning and generate lifelike images and text."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 14f61be5-9701-4ddd-8c26-df7770335bdc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"14f61be5-9701-4ddd-8c26-df7770335bdc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0e3b34c-6620-449f-b66a-4ff56ae82c6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d7db83f3663ada04d53c9f6c94dbd5fc26f849ba45c8d94d4da96501d5ad7087\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9501de56-f184-47ef-bc3f-726e45a32dcf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"398d40d9af064a7b61e611f880535dccf931ce505183b510118a27c58747a58d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"94770c5c-becf-4258-8797-e1ca2b2b126c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fa43d8f6effcc0541c60b7e2482ddef4cc2520021fc3f1ee6d76d3710289f825\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1887, \"end_char_idx\": 5415, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "doc_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "ref_doc_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "chroma:document": "The abstract from \n\"Biological structure and function emerge from scaling unsupervised learning to 250 \nmillion protein sequences\" is\n\n\n*In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised\nlearning has led to major advances in representation learning and statistical generation. In the life sciences, the\nanticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling\nat the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To\nthis end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250\nmillion protein sequences spanning evolutionary diversity. The resulting model contains information about biological\nproperties in its representations. The representations are learned from sequence data alone. The learned representation\nspace has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to\nremote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and\ncan be identified by linear projections. Representation learning produces features that generalize across a range of\napplications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and\nimproving state-of-the-art features for long-range contact prediction.*\n\n\nThe abstract from\n\"Language models of protein sequences at the scale of evolution enable accurate structure prediction\" is\n\n*Large language models have recently been shown to develop emergent capabilities with scale, going beyond\nsimple pattern matching to perform higher level reasoning and generate lifelike images and text. While\nlanguage models trained on protein sequences have been studied at a smaller scale, little is known about\nwhat they learn about biology as they are scaled up. In this work we train models up to 15 billion parameters,\nthe largest language models of proteins to be evaluated to date. We find that as models are scaled they learn\ninformation enabling the prediction of the three-dimensional structure of a protein at the resolution of\nindividual atoms. We present ESMFold for high accuracy end-to-end atomic level structure prediction directly\nfrom the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for\nsequences with low perplexity that are well understood by the language model. ESMFold inference is an\norder of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic\nproteins in practical timescales.*\n\nThe original code can be found [here](https://github.com/facebookresearch/esm) and was\nwas developed by the Fundamental AI Research team at Meta AI.\nESM-1b, ESM-1v and ESM-2 were contributed to huggingface by [jasonliu](https://huggingface.co/jasonliu)\nand [Matt](https://huggingface.co/Rocketknight1).\n\nESMFold was contributed to huggingface by [Matt](https://huggingface.co/Rocketknight1) and\n[Sylvain](https://huggingface.co/sgugger), with a big thank you to Nikita Smetanin, Roshan Rao and Tom Sercu for their\nhelp throughout the process!\n\n## Usage tips\n\n- ESM models are trained with a masked language modeling (MLM) objective.\n- The HuggingFace port of ESMFold uses portions of the [openfold](https://github.com/aqlaboratory/openfold) library. The `openfold` library is licensed under the Apache License 2.0."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 94770c5c-becf-4258-8797-e1ca2b2b126c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"94770c5c-becf-4258-8797-e1ca2b2b126c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0e3b34c-6620-449f-b66a-4ff56ae82c6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d7db83f3663ada04d53c9f6c94dbd5fc26f849ba45c8d94d4da96501d5ad7087\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"14f61be5-9701-4ddd-8c26-df7770335bdc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7845f5ed1badd34f1e737c19839d94ffff46cb34167372c3adacd3c6962d8ca6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3872, \"end_char_idx\": 6570, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "doc_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "ref_doc_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "chroma:document": "In this work we train models up to 15 billion parameters,\nthe largest language models of proteins to be evaluated to date. We find that as models are scaled they learn\ninformation enabling the prediction of the three-dimensional structure of a protein at the resolution of\nindividual atoms. We present ESMFold for high accuracy end-to-end atomic level structure prediction directly\nfrom the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for\nsequences with low perplexity that are well understood by the language model. ESMFold inference is an\norder of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic\nproteins in practical timescales.*\n\nThe original code can be found [here](https://github.com/facebookresearch/esm) and was\nwas developed by the Fundamental AI Research team at Meta AI.\nESM-1b, ESM-1v and ESM-2 were contributed to huggingface by [jasonliu](https://huggingface.co/jasonliu)\nand [Matt](https://huggingface.co/Rocketknight1).\n\nESMFold was contributed to huggingface by [Matt](https://huggingface.co/Rocketknight1) and\n[Sylvain](https://huggingface.co/sgugger), with a big thank you to Nikita Smetanin, Roshan Rao and Tom Sercu for their\nhelp throughout the process!\n\n## Usage tips\n\n- ESM models are trained with a masked language modeling (MLM) objective.\n- The HuggingFace port of ESMFold uses portions of the [openfold](https://github.com/aqlaboratory/openfold) library. The `openfold` library is licensed under the Apache License 2.0.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n## EsmConfig\n\n[[autodoc]] EsmConfig\n    - all\n\n## EsmTokenizer\n\n[[autodoc]] EsmTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## EsmModel\n\n[[autodoc]] EsmModel\n    - forward\n\n## EsmForMaskedLM\n\n[[autodoc]] EsmForMaskedLM\n    - forward\n\n## EsmForSequenceClassification\n\n[[autodoc]] EsmForSequenceClassification\n    - forward\n\n## EsmForTokenClassification\n\n[[autodoc]] EsmForTokenClassification\n    - forward\n\n## EsmForProteinFolding\n\n[[autodoc]] EsmForProteinFolding\n    - forward\n\n</pt>\n<tf>\n\n## TFEsmModel\n\n[[autodoc]] TFEsmModel\n    - call\n\n## TFEsmForMaskedLM\n\n[[autodoc]] TFEsmForMaskedLM\n    - call\n\n## TFEsmForSequenceClassification\n\n[[autodoc]] TFEsmForSequenceClassification\n    - call\n\n## TFEsmForTokenClassification\n\n[[autodoc]] TFEsmForTokenClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a266f8a0-e7f2-4f31-a196-62fcbf76e589
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt", "title": "Pyramid Vision Transformer (PVT)", "tokens": 892, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a266f8a0-e7f2-4f31-a196-62fcbf76e589\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt\", \"title\": \"Pyramid Vision Transformer (PVT)\", \"tokens\": 892, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dae61dce-8f6c-4ac0-837e-4bbd3dd737da\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt\", \"title\": \"Pyramid Vision Transformer (PVT)\", \"tokens\": 892, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e8e7e44c59d35c4df76114714eb0e5267561a2f05094fdc22f06ea678c691588\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3159, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dae61dce-8f6c-4ac0-837e-4bbd3dd737da", "doc_id": "dae61dce-8f6c-4ac0-837e-4bbd3dd737da", "ref_doc_id": "dae61dce-8f6c-4ac0-837e-4bbd3dd737da", "chroma:document": "# Pyramid Vision Transformer (PVT)\n\n## Overview\n\nThe PVT model was proposed in\n[Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://arxiv.org/abs/2102.12122)\nby Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao. The PVT is a type of\nvision transformer that utilizes a pyramid structure to make it an effective backbone for dense prediction tasks. Specifically\nit allows for more fine-grained inputs (4 x 4 pixels per patch) to be used, while simultaneously shrinking the sequence length\nof the Transformer as it deepens - reducing the computational cost. Additionally, a spatial-reduction attention (SRA) layer\nis used to further reduce the resource consumption when learning high-resolution features.\n\nThe abstract from the paper is the following:\n\n*Although convolutional neural networks (CNNs) have achieved great success in computer vision, this work investigates a \nsimpler, convolution-free backbone network useful for many dense prediction tasks. Unlike the recently proposed Vision \nTransformer (ViT) that was designed for image classification specifically, we introduce the Pyramid Vision Transformer \n(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several \nmerits compared to current state of the arts. Different from ViT that typically yields low resolution outputs and \nincurs high computational and memory costs, PVT not only can be trained on dense partitions of an image to achieve high \noutput resolution, which is important for dense prediction, but also uses a progressive shrinking pyramid to reduce the \ncomputations of large feature maps. PVT inherits the advantages of both CNN and Transformer, making it a unified \nbackbone for various vision tasks without convolutions, where it can be used as a direct replacement for CNN backbones. \nWe validate PVT through extensive experiments, showing that it boosts the performance of many downstream tasks, including\nobject detection, instance and semantic segmentation. For example, with a comparable number of parameters, PVT+RetinaNet \nachieves 40.4 AP on the COCO dataset, surpassing ResNet50+RetinNet (36.3 AP) by 4.1 absolute AP (see Figure 2). We hope \nthat PVT could serve as an alternative and useful backbone for pixel-level predictions and facilitate future research.*\n\nThis model was contributed by [Xrenya](https://huggingface.co/Xrenya). The original code can be found [here](https://github.com/whai362/PVT).\n\n\n- PVTv1 on ImageNet-1K\n\n| **Model variant**  |**Size** |**Acc@1**|**Params (M)**|\n|--------------------|:-------:|:-------:|:------------:|\n| PVT-Tiny           |    224  |   75.1  |     13.2     |\n| PVT-Small          |    224  |   79.8  |     24.5     |\n| PVT-Medium         |    224  |   81.2  |     44.2     |\n| PVT-Large          |    224  |   81.7  |     61.4     |\n\n\n## PvtConfig\n\n[[autodoc]] PvtConfig\n\n## PvtImageProcessor\n\n[[autodoc]] PvtImageProcessor\n    - preprocess\n\n## PvtForImageClassification\n\n[[autodoc]] PvtForImageClassification\n    - forward\n\n## PvtModel\n\n[[autodoc]] PvtModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 73248f00-3189-4269-a4b5-3f6cfdaceb15
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"73248f00-3189-4269-a4b5-3f6cfdaceb15\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e4d32f91-d19c-428a-aaa3-d74edf4ac0c5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2871f2b5-bab4-4b31-a3d7-ee9c037b7688\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4f73b6a6ca8e6f396f61c8ee559a3b59dcc31c8f41adc78ed9c513735185d76\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3102, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "ref_doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "chroma:document": "# Reformer\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=reformer\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-reformer-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/reformer-crime-and-punishment\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Reformer model was proposed in the paper [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451.pdf) by Nikita Kitaev, \u0141ukasz Kaiser, Anselm Levskaya.\n\nThe abstract from the paper is the following:\n\n*Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can\nbe prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of\nTransformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its\ncomplexity from O(L^2) to O(Llog(L)), where L is the length of the sequence. Furthermore, we use reversible residual\nlayers instead of the standard residuals, which allows storing activations only once in the training process instead of\nN times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models\nwhile being much more memory-efficient and much faster on long sequences.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be\nfound [here](https://github.com/google/trax/tree/master/trax/models/reformer).\n\n## Usage tips\n\n- Reformer does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035).\n- Use Axial position encoding (see below for more details). It\u2019s a mechanism to avoid having a huge positional encoding matrix (when the sequence length is very big) by factorizing it into smaller matrices.\n- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It\u2019s a technique to avoid computing the full product query-key in the attention layers.\n- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).\n- Compute the feedforward operations by chunks and not on the whole batch.\n\n### Axial Positional Encodings\n\nAxial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)\nand developed by the authors of this model's paper. In models that are treating very long input sequences, the\nconventional position id encodings store an embeddings vector of size \\\\(d\\\\) being the `config.hidden_size` for\nevery position \\\\(i, \\ldots, n_s\\\\), with \\\\(n_s\\\\) being `config.max_embedding_size`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2871f2b5-bab4-4b31-a3d7-ee9c037b7688
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2871f2b5-bab4-4b31-a3d7-ee9c037b7688\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e4d32f91-d19c-428a-aaa3-d74edf4ac0c5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73248f00-3189-4269-a4b5-3f6cfdaceb15\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0c927ebf1bfd9e66ff8c93d617cc23d5b5ef0432b8ac021cf6b67043e1fecbac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"75bac14b-4da7-4f30-8166-ade60f6924bd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0f69baf10066f574a8b8d309e1fb789ea227ec2930c8111245a30fb1f3c43cc7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1493, \"end_char_idx\": 3435, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "ref_doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "chroma:document": "The Authors' code can be\nfound [here](https://github.com/google/trax/tree/master/trax/models/reformer).\n\n## Usage tips\n\n- Reformer does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035).\n- Use Axial position encoding (see below for more details). It\u2019s a mechanism to avoid having a huge positional encoding matrix (when the sequence length is very big) by factorizing it into smaller matrices.\n- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It\u2019s a technique to avoid computing the full product query-key in the attention layers.\n- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).\n- Compute the feedforward operations by chunks and not on the whole batch.\n\n### Axial Positional Encodings\n\nAxial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)\nand developed by the authors of this model's paper. In models that are treating very long input sequences, the\nconventional position id encodings store an embeddings vector of size \\\\(d\\\\) being the `config.hidden_size` for\nevery position \\\\(i, \\ldots, n_s\\\\), with \\\\(n_s\\\\) being `config.max_embedding_size`. This means that having\na sequence length of \\\\(n_s = 2^{19} \\approx 0.5M\\\\) and a `config.hidden_size` of \\\\(d = 2^{10} \\approx 1000\\\\)\nwould result in a position encoding matrix:\n\n$$X_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s\\right]$$\n\nwhich alone has over 500M parameters to store."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 75bac14b-4da7-4f30-8166-ade60f6924bd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"75bac14b-4da7-4f30-8166-ade60f6924bd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e4d32f91-d19c-428a-aaa3-d74edf4ac0c5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2871f2b5-bab4-4b31-a3d7-ee9c037b7688\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9f9085e179402667ff87ba249c075f07ed2d6b879108b32beef1290eaac66a09\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"444c94a2-2e77-4a42-bc12-b30b1e222ae9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"37a387a0577b18f4a59a1f963a979ff706be00179cde55bf3c62d3d399429cd7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1972, \"end_char_idx\": 4393, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "ref_doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "chroma:document": "- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It\u2019s a technique to avoid computing the full product query-key in the attention layers.\n- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).\n- Compute the feedforward operations by chunks and not on the whole batch.\n\n### Axial Positional Encodings\n\nAxial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)\nand developed by the authors of this model's paper. In models that are treating very long input sequences, the\nconventional position id encodings store an embeddings vector of size \\\\(d\\\\) being the `config.hidden_size` for\nevery position \\\\(i, \\ldots, n_s\\\\), with \\\\(n_s\\\\) being `config.max_embedding_size`. This means that having\na sequence length of \\\\(n_s = 2^{19} \\approx 0.5M\\\\) and a `config.hidden_size` of \\\\(d = 2^{10} \\approx 1000\\\\)\nwould result in a position encoding matrix:\n\n$$X_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s\\right]$$\n\nwhich alone has over 500M parameters to store. Axial positional encodings factorize \\\\(X_{i,j}\\\\) into two matrices:\n\n$$X^{1}_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d^1\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s^1\\right]$$\n\nand\n\n$$X^{2}_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d^2\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s^2\\right]$$\n\nwith:\n\n$$d = d^1 + d^2 \\text{ and } n_s = n_s^1 \\times n_s^2 .$$\n\nTherefore the following holds:\n\n$$X_{i,j} = \\begin{cases}\nX^{1}_{i, k}, & \\text{if }\\ i < d^1 \\text{ with } k = j \\mod n_s^1 \\\\\nX^{2}_{i - d^1, l}, & \\text{if } i \\ge d^1 \\text{ with } l = \\lfloor\\frac{j}{n_s^1}\\rfloor\n\\end{cases}$$\n\nIntuitively, this means that a position embedding vector \\\\(x_j \\in \\mathbb{R}^{d}\\\\) is now the composition of two\nfactorized embedding vectors: \\\\(x^1_{k, l} + x^2_{l, k}\\\\), where as the `config.max_embedding_size` dimension\n\\\\(j\\\\) is factorized into \\\\(k \\text{ and } l\\\\). This design ensures that each position embedding vector\n\\\\(x_j\\\\) is unique."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 444c94a2-2e77-4a42-bc12-b30b1e222ae9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"444c94a2-2e77-4a42-bc12-b30b1e222ae9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e4d32f91-d19c-428a-aaa3-d74edf4ac0c5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"75bac14b-4da7-4f30-8166-ade60f6924bd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"05e1045a241961de7e6ffe3f8086bc7c1304f8f449372e4bdfc9d56722da2aa2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0bc78568-1aaa-41c5-af62-71b8fa6f7816\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab830199aa7d0cc38601c1afdbf9c4b39534517ae9843d8fd974688c695455d1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3436, \"end_char_idx\": 5018, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "ref_doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "chroma:document": "Axial positional encodings factorize \\\\(X_{i,j}\\\\) into two matrices:\n\n$$X^{1}_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d^1\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s^1\\right]$$\n\nand\n\n$$X^{2}_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d^2\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s^2\\right]$$\n\nwith:\n\n$$d = d^1 + d^2 \\text{ and } n_s = n_s^1 \\times n_s^2 .$$\n\nTherefore the following holds:\n\n$$X_{i,j} = \\begin{cases}\nX^{1}_{i, k}, & \\text{if }\\ i < d^1 \\text{ with } k = j \\mod n_s^1 \\\\\nX^{2}_{i - d^1, l}, & \\text{if } i \\ge d^1 \\text{ with } l = \\lfloor\\frac{j}{n_s^1}\\rfloor\n\\end{cases}$$\n\nIntuitively, this means that a position embedding vector \\\\(x_j \\in \\mathbb{R}^{d}\\\\) is now the composition of two\nfactorized embedding vectors: \\\\(x^1_{k, l} + x^2_{l, k}\\\\), where as the `config.max_embedding_size` dimension\n\\\\(j\\\\) is factorized into \\\\(k \\text{ and } l\\\\). This design ensures that each position embedding vector\n\\\\(x_j\\\\) is unique.\n\nUsing the above example again, axial position encoding with \\\\(d^1 = 2^9, d^2 = 2^9, n_s^1 = 2^9, n_s^2 = 2^{10}\\\\)\ncan drastically reduced the number of parameters from 500 000 000 to \\\\(2^{18} + 2^{19} \\approx 780 000\\\\) parameters, this means 85% less memory usage.\n\nIn practice, the parameter `config.axial_pos_embds_dim` is set to a tuple \\\\((d^1, d^2)\\\\) which sum has to be\nequal to `config.hidden_size` and `config.axial_pos_shape` is set to a tuple \\\\((n_s^1, n_s^2)\\\\) which\nproduct has to be equal to `config.max_embedding_size`, which during training has to be equal to the *sequence\nlength* of the `input_ids`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0bc78568-1aaa-41c5-af62-71b8fa6f7816
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0bc78568-1aaa-41c5-af62-71b8fa6f7816\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e4d32f91-d19c-428a-aaa3-d74edf4ac0c5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"444c94a2-2e77-4a42-bc12-b30b1e222ae9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ca73e0beccac501b7284d077b1595caf99b989095d60066c8e448ed48d2e644f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"07efaba1-0270-4882-b630-4d3f315d836f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"396889b2d203e7e020062d3dd57b0db89efcffcab854cdbf59464f8e729464b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4317, \"end_char_idx\": 7468, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "ref_doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "chroma:document": "This design ensures that each position embedding vector\n\\\\(x_j\\\\) is unique.\n\nUsing the above example again, axial position encoding with \\\\(d^1 = 2^9, d^2 = 2^9, n_s^1 = 2^9, n_s^2 = 2^{10}\\\\)\ncan drastically reduced the number of parameters from 500 000 000 to \\\\(2^{18} + 2^{19} \\approx 780 000\\\\) parameters, this means 85% less memory usage.\n\nIn practice, the parameter `config.axial_pos_embds_dim` is set to a tuple \\\\((d^1, d^2)\\\\) which sum has to be\nequal to `config.hidden_size` and `config.axial_pos_shape` is set to a tuple \\\\((n_s^1, n_s^2)\\\\) which\nproduct has to be equal to `config.max_embedding_size`, which during training has to be equal to the *sequence\nlength* of the `input_ids`.\n\n\n### LSH Self Attention\n\nIn Locality sensitive hashing (LSH) self attention the key and query projection weights are tied. Therefore, the key\nquery embedding vectors are also tied. LSH self attention uses the locality sensitive hashing mechanism proposed in\n[Practical and Optimal LSH for Angular Distance](https://arxiv.org/abs/1509.02897) to assign each of the tied key\nquery embedding vectors to one of `config.num_buckets` possible buckets. The premise is that the more \"similar\"\nkey query embedding vectors (in terms of *cosine similarity*) are to each other, the more likely they are assigned to\nthe same bucket.\n\nThe accuracy of the LSH mechanism can be improved by increasing `config.num_hashes` or directly the argument\n`num_hashes` of the forward function so that the output of the LSH self attention better approximates the output\nof the \"normal\" full self attention. The buckets are then sorted and chunked into query key embedding vector chunks\neach of length `config.lsh_chunk_length`. For each chunk, the query embedding vectors attend to its key vectors\n(which are tied to themselves) and to the key embedding vectors of `config.lsh_num_chunks_before` previous\nneighboring chunks and `config.lsh_num_chunks_after` following neighboring chunks.\n\nFor more information, see the [original Paper](https://arxiv.org/abs/2001.04451) or this great [blog post](https://www.pragmatic.ml/reformer-deep-dive/).\n\nNote that `config.num_buckets` can also be factorized into a list \\\\((n_{\\text{buckets}}^1,\nn_{\\text{buckets}}^2)\\\\). This way instead of assigning the query key embedding vectors to one of \\\\((1,\\ldots,\nn_{\\text{buckets}})\\\\) they are assigned to one of \\\\((1-1,\\ldots, n_{\\text{buckets}}^1-1, \\ldots,\n1-n_{\\text{buckets}}^2, \\ldots, n_{\\text{buckets}}^1-n_{\\text{buckets}}^2)\\\\). This is crucial for very long sequences to\nsave memory.\n\nWhen training a model from scratch, it is recommended to leave `config.num_buckets=None`, so that depending on the\nsequence length a good value for `num_buckets` is calculated on the fly. This value will then automatically be\nsaved in the config and should be reused for inference.\n\nUsing LSH self attention, the memory and time complexity of the query-key matmul operation can be reduced from\n\\\\(\\mathcal{O}(n_s \\times n_s)\\\\) to \\\\(\\mathcal{O}(n_s \\times \\log(n_s))\\\\), which usually represents the memory\nand time bottleneck in a transformer model, with \\\\(n_s\\\\) being the sequence length."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 07efaba1-0270-4882-b630-4d3f315d836f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"07efaba1-0270-4882-b630-4d3f315d836f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e4d32f91-d19c-428a-aaa3-d74edf4ac0c5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0bc78568-1aaa-41c5-af62-71b8fa6f7816\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2dfbced02b076b3d98a9bd0f3190453540c0815f452c01de69b3334f4d83066b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7471, \"end_char_idx\": 9746, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "ref_doc_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "chroma:document": "### Local Self Attention\n\nLocal self attention is essentially a \"normal\" self attention layer with key, query and value projections, but is\nchunked so that in each chunk of length `config.local_chunk_length` the query embedding vectors only attends to\nthe key embedding vectors in its chunk and to the key embedding vectors of `config.local_num_chunks_before`\nprevious neighboring chunks and `config.local_num_chunks_after` following neighboring chunks.\n\nUsing Local self attention, the memory and time complexity of the query-key matmul operation can be reduced from\n\\\\(\\mathcal{O}(n_s \\times n_s)\\\\) to \\\\(\\mathcal{O}(n_s \\times \\log(n_s))\\\\), which usually represents the memory\nand time bottleneck in a transformer model, with \\\\(n_s\\\\) being the sequence length.\n\n\n### Training\n\nDuring training, we must ensure that the sequence length is set to a value that can be divided by the least common\nmultiple of `config.lsh_chunk_length` and `config.local_chunk_length` and that the parameters of the Axial\nPositional Encodings are correctly set as described above. Reformer is very memory efficient so that the model can\neasily be trained on sequences as long as 64000 tokens.\n\nFor training, the [`ReformerModelWithLMHead`] should be used as follows:\n\n```python\ninput_ids = tokenizer.encode(\"This is a sentence from the training data\", return_tensors=\"pt\")\nloss = model(input_ids, labels=input_ids)[0]\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n## ReformerConfig\n\n[[autodoc]] ReformerConfig\n\n## ReformerTokenizer\n\n[[autodoc]] ReformerTokenizer\n    - save_vocabulary\n\n## ReformerTokenizerFast\n\n[[autodoc]] ReformerTokenizerFast\n\n## ReformerModel\n\n[[autodoc]] ReformerModel\n    - forward\n\n## ReformerModelWithLMHead\n\n[[autodoc]] ReformerModelWithLMHead\n    - forward\n\n## ReformerForMaskedLM\n\n[[autodoc]] ReformerForMaskedLM\n    - forward\n\n## ReformerForSequenceClassification\n\n[[autodoc]] ReformerForSequenceClassification\n    - forward\n\n## ReformerForQuestionAnswering\n\n[[autodoc]] ReformerForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ffbb8ab0-ad32-4915-bdbb-ec6716e7548d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/camembert", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ffbb8ab0-ad32-4915-bdbb-ec6716e7548d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"263828bf-a6e4-4968-bbd9-9e705a921925\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"20e681968b6e7457e163f2013069be4e090b079ef8878d04648e0a71866b5658\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"49de8193-ddc3-43b3-b140-ceba20200f4e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6fe06c8a56f87437b587d671e99662b382783d5338bc58efe976412b0119f52b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1812, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "263828bf-a6e4-4968-bbd9-9e705a921925", "doc_id": "263828bf-a6e4-4968-bbd9-9e705a921925", "ref_doc_id": "263828bf-a6e4-4968-bbd9-9e705a921925", "chroma:document": "# CamemBERT\n\n## Overview\n\nThe CamemBERT model was proposed in [CamemBERT: a Tasty French Language Model](https://arxiv.org/abs/1911.03894) by\n[Louis Martin](https://huggingface.co/louismartin), [Benjamin Muller](https://huggingface.co/benjamin-mlr), [Pedro Javier Ortiz Su\u00e1rez](https://huggingface.co/pjox), Yoann Dupont, Laurent Romary, \u00c9ric Villemonte de la\nClergerie, [Djam\u00e9 Seddah](https://huggingface.co/Djame), and [Beno\u00eet Sagot](https://huggingface.co/sagot). It is based on Facebook's RoBERTa model released in 2019. It is a model\ntrained on 138GB of French text.\n\nThe abstract from the paper is the following:\n\n*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available\nmodels have either been trained on English data or on the concatenation of data in multiple languages. This makes\npractical use of such models --in all languages except English-- very limited. Aiming to address this issue for French,\nwe release CamemBERT, a French version of the Bi-directional Encoders for Transformers (BERT). We measure the\nperformance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging,\ndependency parsing, named-entity recognition, and natural language inference. CamemBERT improves the state of the art\nfor most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and\ndownstream applications for French NLP.*\n\nThis model was contributed by [the ALMAnaCH team (Inria)](https://huggingface.co/almanach). The original code can be found [here](https://camembert-model.fr/).\n\n<Tip>\n\nThis implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well \nas the information relative to the inputs and outputs."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 49de8193-ddc3-43b3-b140-ceba20200f4e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/camembert", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"49de8193-ddc3-43b3-b140-ceba20200f4e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"263828bf-a6e4-4968-bbd9-9e705a921925\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"20e681968b6e7457e163f2013069be4e090b079ef8878d04648e0a71866b5658\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ffbb8ab0-ad32-4915-bdbb-ec6716e7548d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e561fc47685ceaa4e5dfa0dbf78942f7817e6ed4ee8764782756083506a3d68e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 467, \"end_char_idx\": 3589, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "263828bf-a6e4-4968-bbd9-9e705a921925", "doc_id": "263828bf-a6e4-4968-bbd9-9e705a921925", "ref_doc_id": "263828bf-a6e4-4968-bbd9-9e705a921925", "chroma:document": "It is based on Facebook's RoBERTa model released in 2019. It is a model\ntrained on 138GB of French text.\n\nThe abstract from the paper is the following:\n\n*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available\nmodels have either been trained on English data or on the concatenation of data in multiple languages. This makes\npractical use of such models --in all languages except English-- very limited. Aiming to address this issue for French,\nwe release CamemBERT, a French version of the Bi-directional Encoders for Transformers (BERT). We measure the\nperformance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging,\ndependency parsing, named-entity recognition, and natural language inference. CamemBERT improves the state of the art\nfor most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and\ndownstream applications for French NLP.*\n\nThis model was contributed by [the ALMAnaCH team (Inria)](https://huggingface.co/almanach). The original code can be found [here](https://camembert-model.fr/).\n\n<Tip>\n\nThis implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well \nas the information relative to the inputs and outputs.\n\n</Tip>\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## CamembertConfig\n\n[[autodoc]] CamembertConfig\n\n## CamembertTokenizer\n\n[[autodoc]] CamembertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## CamembertTokenizerFast\n\n[[autodoc]] CamembertTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## CamembertModel\n\n[[autodoc]] CamembertModel\n\n## CamembertForCausalLM\n\n[[autodoc]] CamembertForCausalLM\n\n## CamembertForMaskedLM\n\n[[autodoc]] CamembertForMaskedLM\n\n## CamembertForSequenceClassification\n\n[[autodoc]] CamembertForSequenceClassification\n\n## CamembertForMultipleChoice\n\n[[autodoc]] CamembertForMultipleChoice\n\n## CamembertForTokenClassification\n\n[[autodoc]] CamembertForTokenClassification\n\n## CamembertForQuestionAnswering\n\n[[autodoc]] CamembertForQuestionAnswering\n\n</pt>\n<tf>\n\n## TFCamembertModel\n\n[[autodoc]] TFCamembertModel\n\n## TFCamembertForCasualLM\n\n[[autodoc]] TFCamembertForCausalLM\n\n## TFCamembertForMaskedLM\n\n[[autodoc]] TFCamembertForMaskedLM\n\n## TFCamembertForSequenceClassification\n\n[[autodoc]] TFCamembertForSequenceClassification\n\n## TFCamembertForMultipleChoice\n\n[[autodoc]] TFCamembertForMultipleChoice\n\n## TFCamembertForTokenClassification\n\n[[autodoc]] TFCamembertForTokenClassification\n\n## TFCamembertForQuestionAnswering\n\n[[autodoc]] TFCamembertForQuestionAnswering\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fd60fa62-c495-4fb7-8fac-3664936f7903
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese", "title": "GPT-NeoX-Japanese", "tokens": 867, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fd60fa62-c495-4fb7-8fac-3664936f7903\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese\", \"title\": \"GPT-NeoX-Japanese\", \"tokens\": 867, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e9d114f8-8c6d-4ad8-8cd1-c54cede32e9a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese\", \"title\": \"GPT-NeoX-Japanese\", \"tokens\": 867, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41f26dbfc7d9c1657f059fb4f25ab444205b7ebb3354ae530faf9d9ec2cb20c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2461, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e9d114f8-8c6d-4ad8-8cd1-c54cede32e9a", "doc_id": "e9d114f8-8c6d-4ad8-8cd1-c54cede32e9a", "ref_doc_id": "e9d114f8-8c6d-4ad8-8cd1-c54cede32e9a", "chroma:document": "# GPT-NeoX-Japanese\n\n## Overview\n\nWe introduce GPT-NeoX-Japanese, which is an autoregressive language model for Japanese, trained on top of [https://github.com/EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).\nJapanese is a unique language with its large vocabulary and a combination of hiragana, katakana, and kanji writing scripts.\nTo address this distinct structure of the Japanese language, we use a [special sub-word tokenizer](https://github.com/tanreinama/Japanese-BPEEncoder_V2). We are very grateful to *tanreinama* for open-sourcing this incredibly helpful tokenizer.\nFollowing the recommendations from Google's research on [PaLM](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), we have removed bias parameters from transformer blocks, achieving better model performance. Please refer [this article](https://medium.com/ml-abeja/training-a-better-gpt-2-93b157662ae4) in detail.\n\nDevelopment of the model was led by [Shinya Otani](https://github.com/SO0529), [Takayoshi Makabe](https://github.com/spider-man-tm), [Anuj Arora](https://github.com/Anuj040), and [Kyo Hattori](https://github.com/go5paopao) from [ABEJA, Inc.](https://www.abejainc.com/). For more information on this model-building activity, please refer [here (ja)](https://tech-blog.abeja.asia/entry/abeja-gpt-project-202207).\n\n### Usage example\n\nThe `generate()` method can be used to generate text using GPT NeoX Japanese model.\n\n```python\n>>> from transformers import GPTNeoXJapaneseForCausalLM, GPTNeoXJapaneseTokenizer\n\n>>> model = GPTNeoXJapaneseForCausalLM.from_pretrained(\"abeja/gpt-neox-japanese-2.7b\")\n>>> tokenizer = GPTNeoXJapaneseTokenizer.from_pretrained(\"abeja/gpt-neox-japanese-2.7b\")\n\n>>> prompt = \"\u4eba\u3068AI\u304c\u5354\u8abf\u3059\u308b\u305f\u3081\u306b\u306f\u3001\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens, skip_special_tokens=True)[0]\n\n>>> print(gen_text)\n\u4eba\u3068AI\u304c\u5354\u8abf\u3059\u308b\u305f\u3081\u306b\u306f\u3001AI\u3068\u4eba\u304c\u5171\u5b58\u3057\u3001AI\u3092\u6b63\u3057\u304f\u7406\u89e3\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\n```\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## GPTNeoXJapaneseConfig\n\n[[autodoc]] GPTNeoXJapaneseConfig\n\n## GPTNeoXJapaneseTokenizer\n\n[[autodoc]] GPTNeoXJapaneseTokenizer\n\n## GPTNeoXJapaneseModel\n\n[[autodoc]] GPTNeoXJapaneseModel\n    - forward\n\n## GPTNeoXJapaneseForCausalLM\n\n[[autodoc]] GPTNeoXJapaneseForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 30a213c4-d5f4-4021-8eb5-26a3bfc34f91
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mra", "title": "MRA", "tokens": 618, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"30a213c4-d5f4-4021-8eb5-26a3bfc34f91\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mra\", \"title\": \"MRA\", \"tokens\": 618, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09a25eff-c8e4-454c-a80f-19ad85c1cef7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mra\", \"title\": \"MRA\", \"tokens\": 618, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bedabd28fbb4822f335357a25bc84b9467f71bc8b8e004f189d811121c50087d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2004, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09a25eff-c8e4-454c-a80f-19ad85c1cef7", "doc_id": "09a25eff-c8e4-454c-a80f-19ad85c1cef7", "ref_doc_id": "09a25eff-c8e4-454c-a80f-19ad85c1cef7", "chroma:document": "# MRA\n\n## Overview\n\nThe MRA model was proposed in [Multi Resolution Analysis (MRA) for Approximate Self-Attention](https://arxiv.org/abs/2207.10284) by Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh.\n\nThe abstract from the paper is the following:\n\n*Transformers have emerged as a preferred model for many tasks in natural language processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effective ideas include various prespecified sparsity patterns, low-rank basis expansions and combinations thereof. In this paper, we revisit classical Multiresolution Analysis (MRA) concepts such as Wavelets, whose potential value in this setting remains underexplored thus far. We show that simple approximations based on empirical feedback and design choices informed by modern hardware and implementation challenges, eventually yield a MRA-based approach for self-attention with an excellent performance profile across most criteria of interest. We undertake an extensive set of experiments and demonstrate that this multi-resolution scheme outperforms most efficient self-attention proposals and is favorable for both short and long sequences. Code is available at https://github.com/mlpen/mra-attention.*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03).\nThe original code can be found [here](https://github.com/mlpen/mra-attention).\n\n## MraConfig\n\n[[autodoc]] MraConfig\n\n## MraModel\n\n[[autodoc]] MraModel\n    - forward\n\n## MraForMaskedLM\n\n[[autodoc]] MraForMaskedLM\n    - forward\n\n## MraForSequenceClassification\n\n[[autodoc]] MraForSequenceClassification\n    - forward\n\n## MraForMultipleChoice\n\n[[autodoc]] MraForMultipleChoice\n    - forward\n\n## MraForTokenClassification\n\n[[autodoc]] MraForTokenClassification\n    - forward\n\n## MraForQuestionAnswering\n\n[[autodoc]] MraForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4495a41c-3204-465a-9484-c15544d4f646
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4495a41c-3204-465a-9484-c15544d4f646\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ed57a1c-ccd7-4b91-90d2-fac7344d7e10\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b42fecbc-2ce4-440b-8808-2243881c7bd9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"176a40ea098eb50a28e3baf6d0a48e55146e1f8f1b98c3bf3def86b5d3ff85c0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3109, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "ref_doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "chroma:document": "# Pop2Piano\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/spaces/sweetcocoa/pop2piano\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Pop2Piano model was proposed in [Pop2Piano : Pop Audio-based Piano Cover Generation](https://arxiv.org/abs/2211.00895) by Jongho Choi and Kyogu Lee.\n\nPiano covers of pop music are widely enjoyed, but generating them from music is not a trivial task. It requires great \nexpertise with playing piano as well as knowing different characteristics and melodies of a song. With Pop2Piano you \ncan directly generate a cover from a song's audio waveform. It is the first model to directly generate a piano cover \nfrom pop audio without melody and chord extraction modules. \n\nPop2Piano is an encoder-decoder Transformer model based on [T5](https://arxiv.org/pdf/1910.10683.pdf). The input audio \nis transformed to its waveform and passed to the encoder, which transforms it to a latent representation. The decoder \nuses these latent representations to generate token ids in an autoregressive way. Each token id corresponds to one of four \ndifferent token types: time, velocity, note and 'special'. The token ids are then decoded to their equivalent MIDI file.\n\nThe abstract from the paper is the following:\n\n*Piano covers of pop music are enjoyed by many people. However, the\ntask of automatically generating piano covers of pop music is still\nunderstudied. This is partly due to the lack of synchronized\n{Pop, Piano Cover} data pairs, which made it challenging to apply\nthe latest data-intensive deep learning-based methods. To leverage\nthe power of the data-driven approach, we make a large amount of\npaired and synchronized {Pop, Piano Cover} data using an automated\npipeline. In this paper, we present Pop2Piano, a Transformer network\nthat generates piano covers given waveforms of pop music. To the best\nof our knowledge, this is the first model to generate a piano cover\ndirectly from pop audio without using melody and chord extraction\nmodules. We show that Pop2Piano, trained with our dataset, is capable\nof producing plausible piano covers.*\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\nThe original code can be found [here](https://github.com/sweetcocoa/pop2piano).\n\n## Usage tips\n\n* To use Pop2Piano, you will need to install the \ud83e\udd17 Transformers library, as well as the following third party modules:  \n```bash\npip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy\n```\nPlease note that you may need to restart your runtime after installation.\n* Pop2Piano is an Encoder-Decoder based model like T5.\n* Pop2Piano can be used to generate midi-audio files for a given audio sequence.\n* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.\n* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.\n* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b42fecbc-2ce4-440b-8808-2243881c7bd9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b42fecbc-2ce4-440b-8808-2243881c7bd9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ed57a1c-ccd7-4b91-90d2-fac7344d7e10\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4495a41c-3204-465a-9484-c15544d4f646\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ac7403faf52ee7217460c36e340963915fc848e93f3102019d04af1f880aeb4a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e4c2e4d5-015a-4bf8-b9d8-dc306a806761\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fecc9aabea6e4a742ba500b56d7e9862610312e364e7698c0b7d6adf18eca638\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1489, \"end_char_idx\": 4222, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "ref_doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "chroma:document": "This is partly due to the lack of synchronized\n{Pop, Piano Cover} data pairs, which made it challenging to apply\nthe latest data-intensive deep learning-based methods. To leverage\nthe power of the data-driven approach, we make a large amount of\npaired and synchronized {Pop, Piano Cover} data using an automated\npipeline. In this paper, we present Pop2Piano, a Transformer network\nthat generates piano covers given waveforms of pop music. To the best\nof our knowledge, this is the first model to generate a piano cover\ndirectly from pop audio without using melody and chord extraction\nmodules. We show that Pop2Piano, trained with our dataset, is capable\nof producing plausible piano covers.*\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\nThe original code can be found [here](https://github.com/sweetcocoa/pop2piano).\n\n## Usage tips\n\n* To use Pop2Piano, you will need to install the \ud83e\udd17 Transformers library, as well as the following third party modules:  \n```bash\npip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy\n```\nPlease note that you may need to restart your runtime after installation.\n* Pop2Piano is an Encoder-Decoder based model like T5.\n* Pop2Piano can be used to generate midi-audio files for a given audio sequence.\n* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.\n* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.\n* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.\n\n## Examples\n\n- Example using HuggingFace Dataset:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> ds = load_dataset(\"sweetcocoa/pop2piano_ci\", split=\"test\")\n\n>>> inputs = processor(\n...     audio=ds[\"audio\"][0][\"array\"], sampling_rate=ds[\"audio\"][0][\"sampling_rate\"], return_tensors=\"pt\"\n... )\n>>> model_output = model.generate(input_features=inputs[\"input_features\"], composer=\"composer1\")\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"][0]\n>>> tokenizer_output.write(\"./Outputs/midi_output.mid\")\n```\n\n- Example using your own audio file:\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> audio, sr = librosa.load(\"<your_audio_file_here>\", sr=44100)  # feel free to change the sr to a suitable value."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e4c2e4d5-015a-4bf8-b9d8-dc306a806761
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e4c2e4d5-015a-4bf8-b9d8-dc306a806761\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ed57a1c-ccd7-4b91-90d2-fac7344d7e10\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b42fecbc-2ce4-440b-8808-2243881c7bd9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61e90b5a00a803d8834c5008c0b04c1a0c2562f601c647e2d0b7a37cafa33ebd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d298ddb8-792f-4105-a5c9-cc24e4bd1ae2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"98694f1f8b94235607e3f01d18ce2b306e6bd7554ca8122f8467b919d7bdd31b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2633, \"end_char_idx\": 4983, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "ref_doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "chroma:document": "* Pop2Piano is an Encoder-Decoder based model like T5.\n* Pop2Piano can be used to generate midi-audio files for a given audio sequence.\n* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.\n* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.\n* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.\n\n## Examples\n\n- Example using HuggingFace Dataset:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> ds = load_dataset(\"sweetcocoa/pop2piano_ci\", split=\"test\")\n\n>>> inputs = processor(\n...     audio=ds[\"audio\"][0][\"array\"], sampling_rate=ds[\"audio\"][0][\"sampling_rate\"], return_tensors=\"pt\"\n... )\n>>> model_output = model.generate(input_features=inputs[\"input_features\"], composer=\"composer1\")\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"][0]\n>>> tokenizer_output.write(\"./Outputs/midi_output.mid\")\n```\n\n- Example using your own audio file:\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> audio, sr = librosa.load(\"<your_audio_file_here>\", sr=44100)  # feel free to change the sr to a suitable value.\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = processor(audio=audio, sampling_rate=sr, return_tensors=\"pt\")\n>>> model_output = model.generate(input_features=inputs[\"input_features\"], composer=\"composer1\")\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"][0]\n>>> tokenizer_output.write(\"./Outputs/midi_output.mid\")\n```\n\n- Example of processing multiple audio files in batch:\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> # feel free to change the sr to a suitable value."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d298ddb8-792f-4105-a5c9-cc24e4bd1ae2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d298ddb8-792f-4105-a5c9-cc24e4bd1ae2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ed57a1c-ccd7-4b91-90d2-fac7344d7e10\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4c2e4d5-015a-4bf8-b9d8-dc306a806761\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7bdc11e137df517f063d2165c08dc3b3ec6c40c441ff9cae9eec0f12225fb327\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"81e96e8a-6e27-41f9-b97a-7337ba9d2e53\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d111db0729e3d39edcd4585a8418407c6ab6d2c475215c82eb65fe86b5fc4bbe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4223, \"end_char_idx\": 7426, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "ref_doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "chroma:document": ">>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = processor(audio=audio, sampling_rate=sr, return_tensors=\"pt\")\n>>> model_output = model.generate(input_features=inputs[\"input_features\"], composer=\"composer1\")\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"][0]\n>>> tokenizer_output.write(\"./Outputs/midi_output.mid\")\n```\n\n- Example of processing multiple audio files in batch:\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> # feel free to change the sr to a suitable value.\n>>> audio1, sr1 = librosa.load(\"<your_first_audio_file_here>\", sr=44100)  \n>>> audio2, sr2 = librosa.load(\"<your_second_audio_file_here>\", sr=44100)\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = processor(audio=[audio1, audio2], sampling_rate=[sr1, sr2], return_attention_mask=True, return_tensors=\"pt\")\n>>> # Since we now generating in batch(2 audios) we must pass the attention_mask\n>>> model_output = model.generate(\n...     input_features=inputs[\"input_features\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     composer=\"composer1\",\n... )\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"]\n\n>>> # Since we now have 2 generated MIDI files\n>>> tokenizer_output[0].write(\"./Outputs/midi_output1.mid\")\n>>> tokenizer_output[1].write(\"./Outputs/midi_output2.mid\")\n```\n\n\n- Example of processing multiple audio files in batch (Using `Pop2PianoFeatureExtractor` and `Pop2PianoTokenizer`):\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoFeatureExtractor, Pop2PianoTokenizer\n\n>>> # feel free to change the sr to a suitable value.\n>>> audio1, sr1 = librosa.load(\"<your_first_audio_file_here>\", sr=44100)  \n>>> audio2, sr2 = librosa.load(\"<your_second_audio_file_here>\", sr=44100)\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> feature_extractor = Pop2PianoFeatureExtractor.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> tokenizer = Pop2PianoTokenizer.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = feature_extractor(\n...     audio=[audio1, audio2], \n...     sampling_rate=[sr1, sr2], \n...     return_attention_mask=True, \n...     return_tensors=\"pt\",\n... )\n>>> # Since we now generating in batch(2 audios) we must pass the attention_mask\n>>> model_output = model.generate(\n...     input_features=inputs[\"input_features\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     composer=\"composer1\",\n... )\n>>> tokenizer_output = tokenizer.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"]\n\n>>> # Since we now have 2 generated MIDI files\n>>> tokenizer_output[0].write(\"./Outputs/midi_output1.mid\")\n>>> tokenizer_output[1].write(\"./Outputs/midi_output2.mid\")\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 81e96e8a-6e27-41f9-b97a-7337ba9d2e53
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"81e96e8a-6e27-41f9-b97a-7337ba9d2e53\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ed57a1c-ccd7-4b91-90d2-fac7344d7e10\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d298ddb8-792f-4105-a5c9-cc24e4bd1ae2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9377663cf869f027332ede595313b64663756fd9e224c07d3b5b5d9bc3035261\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5981, \"end_char_idx\": 7814, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "ref_doc_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "chroma:document": "- Example of processing multiple audio files in batch (Using `Pop2PianoFeatureExtractor` and `Pop2PianoTokenizer`):\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoFeatureExtractor, Pop2PianoTokenizer\n\n>>> # feel free to change the sr to a suitable value.\n>>> audio1, sr1 = librosa.load(\"<your_first_audio_file_here>\", sr=44100)  \n>>> audio2, sr2 = librosa.load(\"<your_second_audio_file_here>\", sr=44100)\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> feature_extractor = Pop2PianoFeatureExtractor.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> tokenizer = Pop2PianoTokenizer.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = feature_extractor(\n...     audio=[audio1, audio2], \n...     sampling_rate=[sr1, sr2], \n...     return_attention_mask=True, \n...     return_tensors=\"pt\",\n... )\n>>> # Since we now generating in batch(2 audios) we must pass the attention_mask\n>>> model_output = model.generate(\n...     input_features=inputs[\"input_features\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     composer=\"composer1\",\n... )\n>>> tokenizer_output = tokenizer.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"]\n\n>>> # Since we now have 2 generated MIDI files\n>>> tokenizer_output[0].write(\"./Outputs/midi_output1.mid\")\n>>> tokenizer_output[1].write(\"./Outputs/midi_output2.mid\")\n```\n\n\n## Pop2PianoConfig\n\n[[autodoc]] Pop2PianoConfig\n\n## Pop2PianoFeatureExtractor\n\n[[autodoc]] Pop2PianoFeatureExtractor\n    - __call__\n\n## Pop2PianoForConditionalGeneration\n\n[[autodoc]] Pop2PianoForConditionalGeneration\n    - forward\n    - generate\n\n## Pop2PianoTokenizer\n\n[[autodoc]] Pop2PianoTokenizer\n    - __call__\n\n## Pop2PianoProcessor\n\n[[autodoc]] Pop2PianoProcessor\n    - __call__"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 59c91e0f-8a44-4116-9b3a-0f05b24df54e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/falcon_mamba", "title": "FalconMamba", "tokens": 1178, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"59c91e0f-8a44-4116-9b3a-0f05b24df54e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/falcon_mamba\", \"title\": \"FalconMamba\", \"tokens\": 1178, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c04a1c8-6597-4009-b945-4a679ae3869d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/falcon_mamba\", \"title\": \"FalconMamba\", \"tokens\": 1178, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9486051287f6a75b59f9704b8f4b2fd89d0faef074025d4694e77b3fb9d0759e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"de9e4393-d0cb-4e23-b328-7bbb1a92d79e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"66d91ab626ece513a4cf1cee0315161574ada194ef8049084d21a6147b7b6218\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3193, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c04a1c8-6597-4009-b945-4a679ae3869d", "doc_id": "5c04a1c8-6597-4009-b945-4a679ae3869d", "ref_doc_id": "5c04a1c8-6597-4009-b945-4a679ae3869d", "chroma:document": "# FalconMamba\n\n## Overview\n\nThe FalconMamba model was proposed by TII UAE (Technology Innovation Institute) in their release.\n\nThe abstract from the paper is the following:\n\n*We present FalconMamba, a new base large language model based on the novel Mamba architecture. FalconMamba is trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure Mamba-based model, FalconMamba surpasses leading open-weight models based on Transformers, such as Mistral 7B, Llama3 8B, and Falcon2 11B. It is on par with Gemma 7B and outperforms models with different architecture designs, such as RecurrentGemma 9B. Currently, FalconMamba is the best-performing Mamba model in the literature at this scale, surpassing both existing Mamba and hybrid Mamba-Transformer models.\nDue to its architecture, FalconMamba is significantly faster at inference and requires substantially less memory for long sequence generation. Despite recent studies suggesting that hybrid Mamba-Transformer models outperform pure architecture designs, we argue and demonstrate that the pure Mamba design can achieve similar, even superior results compared to the hybrid design. We make the weights of our implementation of FalconMamba publicly available under a permissive license.*\n\nTips:\n\n- FalconMamba is mostly based on Mamba architecutre, the same [tips and best practices](./mamba) would be relevant here.\n\nThe model has been trained on approximtely 6T tokens consisting a mixture of many data sources such as RefineWeb, Cosmopedia and Math data.\n\nFor more details about the training procedure and the architecture, have a look at [the technical paper of FalconMamba]() (coming soon).\n\n# Usage\n\nBelow we demonstrate how to use the model:\n\n```python \nfrom transformers import FalconMambaForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"tiiuae/falcon-mamba-7b\")\nmodel = FalconMambaForCausalLM.from_pretrained(\"tiiuae/falcon-mamba-7b\")\n\ninput_ids = tokenizer(\"Hey how are you doing?\", return_tensors= \"pt\")[\"input_ids\"]\n\nout = model.generate(input_ids, max_new_tokens=10)\nprint(tokenizer.batch_decode(out))\n```\n\nThe architecture is also compatible with `torch.compile` for faster generation:\n\n```python \nfrom transformers import FalconMambaForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"tiiuae/falcon-mamba-7b\")\nmodel = FalconMambaForCausalLM.from_pretrained(\"tiiuae/falcon-mamba-7b\", torch_dtype=torch.bfloat16).to(0)\nmodel = torch.compile(model)\n\ninput_ids = tokenizer(\"Hey how are you doing?\", return_tensors= \"pt\")[\"input_ids\"]\n\nout = model.generate(input_ids, max_new_tokens=10)\nprint(tokenizer.batch_decode(out))\n```\n\nIf you have access to a GPU that is compatible with `bitsandbytes`, you can also quantize the model in 4-bit precision:\n\n```python \nfrom transformers import FalconMambaForCausalLM, AutoTokenizer, BitsAndBytesConfig\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"tiiuae/falcon-mamba-7b\")\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\nmodel = FalconMambaForCausalLM.from_pretrained(\"tiiuae/falcon-mamba-7b\", quantization_config=quantization_config)\n\ninput_ids = tokenizer(\"Hey how are you doing?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: de9e4393-d0cb-4e23-b328-7bbb1a92d79e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/falcon_mamba", "title": "FalconMamba", "tokens": 1178, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"de9e4393-d0cb-4e23-b328-7bbb1a92d79e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/falcon_mamba\", \"title\": \"FalconMamba\", \"tokens\": 1178, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c04a1c8-6597-4009-b945-4a679ae3869d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/falcon_mamba\", \"title\": \"FalconMamba\", \"tokens\": 1178, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9486051287f6a75b59f9704b8f4b2fd89d0faef074025d4694e77b3fb9d0759e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"59c91e0f-8a44-4116-9b3a-0f05b24df54e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/falcon_mamba\", \"title\": \"FalconMamba\", \"tokens\": 1178, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61664343b410738fbd42954652e79e635db8ee867315a67b846831d2ffcd54c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1669, \"end_char_idx\": 4243, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c04a1c8-6597-4009-b945-4a679ae3869d", "doc_id": "5c04a1c8-6597-4009-b945-4a679ae3869d", "ref_doc_id": "5c04a1c8-6597-4009-b945-4a679ae3869d", "chroma:document": "# Usage\n\nBelow we demonstrate how to use the model:\n\n```python \nfrom transformers import FalconMambaForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"tiiuae/falcon-mamba-7b\")\nmodel = FalconMambaForCausalLM.from_pretrained(\"tiiuae/falcon-mamba-7b\")\n\ninput_ids = tokenizer(\"Hey how are you doing?\", return_tensors= \"pt\")[\"input_ids\"]\n\nout = model.generate(input_ids, max_new_tokens=10)\nprint(tokenizer.batch_decode(out))\n```\n\nThe architecture is also compatible with `torch.compile` for faster generation:\n\n```python \nfrom transformers import FalconMambaForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"tiiuae/falcon-mamba-7b\")\nmodel = FalconMambaForCausalLM.from_pretrained(\"tiiuae/falcon-mamba-7b\", torch_dtype=torch.bfloat16).to(0)\nmodel = torch.compile(model)\n\ninput_ids = tokenizer(\"Hey how are you doing?\", return_tensors= \"pt\")[\"input_ids\"]\n\nout = model.generate(input_ids, max_new_tokens=10)\nprint(tokenizer.batch_decode(out))\n```\n\nIf you have access to a GPU that is compatible with `bitsandbytes`, you can also quantize the model in 4-bit precision:\n\n```python \nfrom transformers import FalconMambaForCausalLM, AutoTokenizer, BitsAndBytesConfig\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"tiiuae/falcon-mamba-7b\")\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\nmodel = FalconMambaForCausalLM.from_pretrained(\"tiiuae/falcon-mamba-7b\", quantization_config=quantization_config)\n\ninput_ids = tokenizer(\"Hey how are you doing?\", return_tensors= \"pt\")[\"input_ids\"]\n\nout = model.generate(input_ids, max_new_tokens=10)\nprint(tokenizer.batch_decode(out))\n```\n\nYou can also play with the instruction fine-tuned model:\n\n```python \nfrom transformers import FalconMambaForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"tiiuae/falcon-mamba-7b-instruct\")\nmodel = FalconMambaForCausalLM.from_pretrained(\"tiiuae/falcon-mamba-7b-instruct\")\n\n# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating\nmessages = [\n    {\"role\": \"user\", \"content\": \"How many helicopters can a human eat in one sitting?\"},\n]\ninput_ids = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True).input_ids\n\noutputs = model.generate(input_ids)\nprint(tokenizer.decode(outputs[0]))\n```\n\n## FalconMambaConfig\n\n[[autodoc]] FalconMambaConfig\n\n## FalconMambaModel\n\n[[autodoc]] FalconMambaModel\n    - forward\n\n## FalconMambaLMHeadModel\n\n[[autodoc]] FalconMambaForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b8e45573-fd81-472b-9120-e398582ff9d9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convnextv2", "title": "ConvNeXt V2", "tokens": 966, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b8e45573-fd81-472b-9120-e398582ff9d9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6a5b39cd-bb67-4f90-842e-02ab973e3e13\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f0ca1f2fb4c6a3f6e12095f0883d1cf488e8eeb8c2aabeeba3a4a1a7cce7c71d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f80ce2f1-dedf-4804-89fb-88b8fe8f7c40\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"079ce304f51d159f26bd7a0e31e7fde21db4509f17d12205709b7e679b34f79f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3246, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6a5b39cd-bb67-4f90-842e-02ab973e3e13", "doc_id": "6a5b39cd-bb67-4f90-842e-02ab973e3e13", "ref_doc_id": "6a5b39cd-bb67-4f90-842e-02ab973e3e13", "chroma:document": "# ConvNeXt V2\n\n## Overview\n\nThe ConvNeXt V2 model was proposed in [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://arxiv.org/abs/2301.00808) by Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, Saining Xie.\nConvNeXt V2 is a pure convolutional model (ConvNet), inspired by the design of Vision Transformers, and a successor of [ConvNeXT](convnext).\n\nThe abstract from the paper is the following:\n\n*Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt, have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked  autoencoders (MAE). However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnextv2_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ConvNeXt V2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2301.00808\">original paper</a>.</small>\n\nThis model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/facebookresearch/ConvNeXt-V2).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ConvNeXt V2.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ConvNextV2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ConvNextV2Config\n\n[[autodoc]] ConvNextV2Config\n\n## ConvNextV2Model\n\n[[autodoc]] ConvNextV2Model\n    - forward\n\n## ConvNextV2ForImageClassification\n\n[[autodoc]] ConvNextV2ForImageClassification\n    - forward\n\n## TFConvNextV2Model\n\n[[autodoc]] TFConvNextV2Model\n    - call"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f80ce2f1-dedf-4804-89fb-88b8fe8f7c40
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convnextv2", "title": "ConvNeXt V2", "tokens": 966, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f80ce2f1-dedf-4804-89fb-88b8fe8f7c40\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6a5b39cd-bb67-4f90-842e-02ab973e3e13\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f0ca1f2fb4c6a3f6e12095f0883d1cf488e8eeb8c2aabeeba3a4a1a7cce7c71d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b8e45573-fd81-472b-9120-e398582ff9d9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b9665f3270c2d456742964473b19df33a0024f567e33d1a2f1f12c5e88588725\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3249, \"end_char_idx\": 3345, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6a5b39cd-bb67-4f90-842e-02ab973e3e13", "doc_id": "6a5b39cd-bb67-4f90-842e-02ab973e3e13", "ref_doc_id": "6a5b39cd-bb67-4f90-842e-02ab973e3e13", "chroma:document": "## TFConvNextV2ForImageClassification\n\n[[autodoc]] TFConvNextV2ForImageClassification\n    - call"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 390f8236-91ee-4d90-830d-c5b6ca98b401
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"390f8236-91ee-4d90-830d-c5b6ca98b401\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"18675c29-2a73-421d-a251-50a7e7d2b784\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0c2b72708a88e42348ba6008b0904a2e5df325131cc94aca277adcc4b6982\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd7f8da0-b167-46ef-a058-b4c31571b9d2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5150e208065b58a2c9c3756a9d5c79d3cca0f3d2dee1f7f52a31e78e77a112d8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3237, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "ref_doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "chroma:document": "# Donut\n\n## Overview\n\nThe Donut model was proposed in [OCR-free Document Understanding Transformer](https://arxiv.org/abs/2111.15664) by\nGeewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, Seunghyun Park.\nDonut consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform document understanding\ntasks such as document image classification, form understanding and visual question answering.\n\nThe abstract from the paper is the following:\n\n*Understanding document images (e.g., invoices) is a core but challenging task since it requires complex functions such as reading text and a holistic understanding of the document. Current Visual Document Understanding (VDU) methods outsource the task of reading text to off-the-shelf Optical Character Recognition (OCR) engines and focus on the understanding task with the OCR outputs. Although such OCR-based approaches have shown promising performance, they suffer from 1) high computational costs for using OCR; 2) inflexibility of OCR models on languages or types of document; 3) OCR error propagation to the subsequent process. To address these issues, in this paper, we introduce a novel OCR-free VDU model named Donut, which stands for Document understanding transformer. As the first step in OCR-free VDU research, we propose a simple architecture (i.e., Transformer) with a pre-training objective (i.e., cross-entropy loss). Donut is conceptually simple yet effective. Through extensive experiments and analyses, we show a simple OCR-free VDU model, Donut, achieves state-of-the-art performances on various VDU tasks in terms of both speed and accuracy. In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/donut_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Donut high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2111.15664\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/clovaai/donut).\n\n## Usage tips\n\n- The quickest way to get started with Donut is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Donut), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- Donut is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.\n\n## Inference examples\n\nDonut's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`DonutImageProcessor`] class is responsible for preprocessing the input image and\n[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`] decodes the generated target tokens to the target string. The\n[`DonutProcessor`] wraps [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]\ninto a single instance to both extract the input features and decode the predicted token ids."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dd7f8da0-b167-46ef-a058-b4c31571b9d2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dd7f8da0-b167-46ef-a058-b4c31571b9d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"18675c29-2a73-421d-a251-50a7e7d2b784\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0c2b72708a88e42348ba6008b0904a2e5df325131cc94aca277adcc4b6982\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"390f8236-91ee-4d90-830d-c5b6ca98b401\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e219688d14b8face5cd23400ac813d08b253021f4ae937652f206c35218d721b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6c5c125a-a3e8-44f8-8309-8423f683c9e8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ebaa59bff908fade149153d343a38d8556911e98881af4ee450717dd2e82be83\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1715, \"end_char_idx\": 4745, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "ref_doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "chroma:document": "In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/donut_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Donut high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2111.15664\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/clovaai/donut).\n\n## Usage tips\n\n- The quickest way to get started with Donut is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Donut), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- Donut is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.\n\n## Inference examples\n\nDonut's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`DonutImageProcessor`] class is responsible for preprocessing the input image and\n[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`] decodes the generated target tokens to the target string. The\n[`DonutProcessor`] wraps [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step Document Image Classification\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-rvlcdip\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-rvlcdip\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[1][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_rvlcdip>\"\n>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6c5c125a-a3e8-44f8-8309-8423f683c9e8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6c5c125a-a3e8-44f8-8309-8423f683c9e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"18675c29-2a73-421d-a251-50a7e7d2b784\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0c2b72708a88e42348ba6008b0904a2e5df325131cc94aca277adcc4b6982\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd7f8da0-b167-46ef-a058-b4c31571b9d2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"aeb44d94f35f1354e2e086a991e98e0ff58a228ba0d241edb91fc0745bd337b2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2fa385f1-2e81-4683-bcd0-8c7ae015f5fe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0ae56e88d13670c9a9dcff7cad024723ecc5c51f867ae4b1dbfad946f98c5b9e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3239, \"end_char_idx\": 6382, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "ref_doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "chroma:document": "- Step-by-step Document Image Classification\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-rvlcdip\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-rvlcdip\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[1][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_rvlcdip>\"\n>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<.*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'class': 'advertisement'}\n```\n\n- Step-by-step Document Parsing\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-cord-v2\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-cord-v2\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[2][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_cord-v2>\"\n>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2fa385f1-2e81-4683-bcd0-8c7ae015f5fe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2fa385f1-2e81-4683-bcd0-8c7ae015f5fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"18675c29-2a73-421d-a251-50a7e7d2b784\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0c2b72708a88e42348ba6008b0904a2e5df325131cc94aca277adcc4b6982\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6c5c125a-a3e8-44f8-8309-8423f683c9e8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0e8e851e8d9f6893817ef7e5ef74ae06abf82564d5842cd9e466f74b2febe6f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"759563f7-02e8-491b-a2e2-9cfb3c3f8f98\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"44d407268a4750aa1975cf042079097ed14e1cf00ef0b8c79213157a6c4e935e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4745, \"end_char_idx\": 7541, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "ref_doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "chroma:document": "*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'class': 'advertisement'}\n```\n\n- Step-by-step Document Parsing\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-cord-v2\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-cord-v2\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[2][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_cord-v2>\"\n>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<.*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'menu': {'nm': 'CINNAMON SUGAR', 'unitprice': '17,000', 'cnt': '1 x', 'price': '17,000'}, 'sub_total': {'subtotal_price': '17,000'}, 'total': {'total_price': '17,000', 'cashprice': '20,000', 'changeprice': '3,000'}}\n```\n\n- Step-by-step Document Visual Question Answering (DocVQA)\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-docvqa\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-docvqa\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image from the DocVQA dataset\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[0][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_docvqa><s_question>{user_input}</s_question><s_answer>\"\n>>> question = \"When is the coffee break?\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 759563f7-02e8-491b-a2e2-9cfb3c3f8f98
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"759563f7-02e8-491b-a2e2-9cfb3c3f8f98\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"18675c29-2a73-421d-a251-50a7e7d2b784\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0c2b72708a88e42348ba6008b0904a2e5df325131cc94aca277adcc4b6982\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2fa385f1-2e81-4683-bcd0-8c7ae015f5fe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"664765be61037ce74b7fcd3c2d2024cb097d493c16494e7bd057298b1bdc19d8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6382, \"end_char_idx\": 9209, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "ref_doc_id": "18675c29-2a73-421d-a251-50a7e7d2b784", "chroma:document": "*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'menu': {'nm': 'CINNAMON SUGAR', 'unitprice': '17,000', 'cnt': '1 x', 'price': '17,000'}, 'sub_total': {'subtotal_price': '17,000'}, 'total': {'total_price': '17,000', 'cashprice': '20,000', 'changeprice': '3,000'}}\n```\n\n- Step-by-step Document Visual Question Answering (DocVQA)\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-docvqa\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-docvqa\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image from the DocVQA dataset\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[0][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_docvqa><s_question>{user_input}</s_question><s_answer>\"\n>>> question = \"When is the coffee break?\"\n>>> prompt = task_prompt.replace(\"{user_input}\", question)\n>>> decoder_input_ids = processor.tokenizer(prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<.*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'question': 'When is the coffee break?', 'answer': '11-14 to 11:39 a.m.'}\n```\n\nSee the [model hub](https://huggingface.co/models?filter=donut) to look for Donut checkpoints.\n\n## Training\n\nWe refer to the [tutorial notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Donut).\n\n## DonutSwinConfig\n\n[[autodoc]] DonutSwinConfig\n\n## DonutImageProcessor\n\n[[autodoc]] DonutImageProcessor\n    - preprocess\n\n## DonutFeatureExtractor\n\n[[autodoc]] DonutFeatureExtractor\n    - __call__\n\n## DonutProcessor\n\n[[autodoc]] DonutProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## DonutSwinModel\n\n[[autodoc]] DonutSwinModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b95ad636-5e71-43c6-90f3-ef79db75f20d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mluke", "title": "mLUKE", "tokens": 722, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b95ad636-5e71-43c6-90f3-ef79db75f20d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mluke\", \"title\": \"mLUKE\", \"tokens\": 722, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eebfdd1b-f09d-4c0e-9459-9f1cd79919ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mluke\", \"title\": \"mLUKE\", \"tokens\": 722, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b7ef0fbe422b03665261c8ad99eeefca8f42f4dc2cfd53b2537d6e5add7464e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2471, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eebfdd1b-f09d-4c0e-9459-9f1cd79919ee", "doc_id": "eebfdd1b-f09d-4c0e-9459-9f1cd79919ee", "ref_doc_id": "eebfdd1b-f09d-4c0e-9459-9f1cd79919ee", "chroma:document": "# mLUKE\n\n## Overview\n\nThe mLUKE model was proposed in [mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models](https://arxiv.org/abs/2110.08151) by Ryokan Ri, Ikuya Yamada, and Yoshimasa Tsuruoka. It's a multilingual extension\nof the [LUKE model](https://arxiv.org/abs/2010.01057) trained on the basis of XLM-RoBERTa.\n\nIt is based on XLM-RoBERTa and adds entity embeddings, which helps improve performance on various downstream tasks\ninvolving reasoning about entities such as named entity recognition, extractive question answering, relation\nclassification, cloze-style knowledge completion.\n\nThe abstract from the paper is the following:\n\n*Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual\nalignment information from Wikipedia entities. However, existing methods only exploit entity information in pretraining\nand do not explicitly use entities in downstream tasks. In this study, we explore the effectiveness of leveraging\nentity representations for downstream cross-lingual tasks. We train a multilingual language model with 24 languages\nwith entity representations and show the model consistently outperforms word-based pretrained models in various\ncross-lingual transfer tasks. We also analyze the model and the key insight is that incorporating entity\nrepresentations into the input allows us to extract more language-agnostic features. We also evaluate the model with a\nmultilingual cloze prompt task with the mLAMA dataset. We show that entity-based prompt elicits correct factual\nknowledge more likely than using only word representations.*\n\nThis model was contributed by [ryo0634](https://huggingface.co/ryo0634). The original code can be found [here](https://github.com/studio-ousia/luke).\n\n## Usage tips\n\nOne can directly plug in the weights of mLUKE into a LUKE model, like so:\n\n```python\nfrom transformers import LukeModel\n\nmodel = LukeModel.from_pretrained(\"studio-ousia/mluke-base\")\n```\n\nNote that mLUKE has its own tokenizer, [`MLukeTokenizer`]. You can initialize it as follows:\n\n```python\nfrom transformers import MLukeTokenizer\n\ntokenizer = MLukeTokenizer.from_pretrained(\"studio-ousia/mluke-base\")\n```\n\n<Tip>\n\nAs mLUKE's architecture is equivalent to that of LUKE, one can refer to [LUKE's documentation page](luke) for all\ntips, code examples and notebooks.\n\n</Tip>\n\n## MLukeTokenizer\n\n[[autodoc]] MLukeTokenizer\n    - __call__\n    - save_vocabulary"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bb1aa1fc-273a-4892-b333-e846d398b06f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bb1aa1fc-273a-4892-b333-e846d398b06f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84b1b074-3ecb-4005-8b6b-d264005c6912\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"65261afed7a170b012df35f61d1feaf43620c52efd42dbfb41212add7ba6d6ee\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"87d1aba4-44b7-4919-a84b-8c80c77b3be0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d02c7793a966c9e1c22ced7965c51a1d9845be5fcfb0e57cf219e9f028a4fd01\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3038, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "doc_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "ref_doc_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "chroma:document": "# QDQBERT\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe QDQBERT model can be referenced in [Integer Quantization for Deep Learning Inference: Principles and Empirical\nEvaluation](https://arxiv.org/abs/2004.09602) by Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev and Paulius\nMicikevicius.\n\nThe abstract from the paper is the following:\n\n*Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by\ntaking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of\nquantization parameters and evaluate their choices on a wide range of neural network models for different application\ndomains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration\nby processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is\nable to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are\nmore difficult to quantize, such as MobileNets and BERT-large.*\n\nThis model was contributed by [shangz](https://huggingface.co/shangz).\n\n## Usage tips\n\n- QDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to (i) linear layer\n  inputs and weights, (ii) matmul inputs, (iii) residual add inputs, in BERT model.\n- QDQBERT requires the dependency of [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). To install `pip install pytorch-quantization --extra-index-url https://pypi.ngc.nvidia.com`\n- QDQBERT model can be loaded from any checkpoint of HuggingFace BERT model (for example *google-bert/bert-base-uncased*), and\n  perform Quantization Aware Training/Post Training Quantization.\n- A complete example of using QDQBERT model to perform Quatization Aware Training and Post Training Quantization for\n  SQUAD task can be found at [transformers/examples/research_projects/quantization-qdqbert/](examples/research_projects/quantization-qdqbert/).\n\n### Set default quantizers\n\nQDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to BERT by\n`TensorQuantizer` in [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). `TensorQuantizer` is the module\nfor quantizing tensors, with `QuantDescriptor` defining how the tensor should be quantized. Refer to [Pytorch\nQuantization Toolkit userguide](https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/userguide.html) for more details.\n\nBefore creating QDQBERT model, one has to set the default `QuantDescriptor` defining default tensor quantizers."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 87d1aba4-44b7-4919-a84b-8c80c77b3be0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"87d1aba4-44b7-4919-a84b-8c80c77b3be0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84b1b074-3ecb-4005-8b6b-d264005c6912\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"65261afed7a170b012df35f61d1feaf43620c52efd42dbfb41212add7ba6d6ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bb1aa1fc-273a-4892-b333-e846d398b06f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"634afcb0283a261f5d5da1f188ce818d6faa4a65e75fe27594aad795cc1603ce\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"81c7152a-ae7b-419f-975c-44262b3336ef\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b0f3520e3bc68e6f7e18b8fb72f58f64bebf58b5a7697edc39fa8cdffd565b90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1471, \"end_char_idx\": 4746, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "doc_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "ref_doc_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "chroma:document": "## Usage tips\n\n- QDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to (i) linear layer\n  inputs and weights, (ii) matmul inputs, (iii) residual add inputs, in BERT model.\n- QDQBERT requires the dependency of [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). To install `pip install pytorch-quantization --extra-index-url https://pypi.ngc.nvidia.com`\n- QDQBERT model can be loaded from any checkpoint of HuggingFace BERT model (for example *google-bert/bert-base-uncased*), and\n  perform Quantization Aware Training/Post Training Quantization.\n- A complete example of using QDQBERT model to perform Quatization Aware Training and Post Training Quantization for\n  SQUAD task can be found at [transformers/examples/research_projects/quantization-qdqbert/](examples/research_projects/quantization-qdqbert/).\n\n### Set default quantizers\n\nQDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to BERT by\n`TensorQuantizer` in [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). `TensorQuantizer` is the module\nfor quantizing tensors, with `QuantDescriptor` defining how the tensor should be quantized. Refer to [Pytorch\nQuantization Toolkit userguide](https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/userguide.html) for more details.\n\nBefore creating QDQBERT model, one has to set the default `QuantDescriptor` defining default tensor quantizers.\n\nExample:\n\n```python\n>>> import pytorch_quantization.nn as quant_nn\n>>> from pytorch_quantization.tensor_quant import QuantDescriptor\n\n>>> # The default tensor quantizer is set to use Max calibration method\n>>> input_desc = QuantDescriptor(num_bits=8, calib_method=\"max\")\n>>> # The default tensor quantizer is set to be per-channel quantization for weights\n>>> weight_desc = QuantDescriptor(num_bits=8, axis=((0,)))\n>>> quant_nn.QuantLinear.set_default_quant_desc_input(input_desc)\n>>> quant_nn.QuantLinear.set_default_quant_desc_weight(weight_desc)\n```\n\n### Calibration\n\nCalibration is the terminology of passing data samples to the quantizer and deciding the best scaling factors for\ntensors. After setting up the tensor quantizers, one can use the following example to calibrate the model:\n\n```python\n>>> # Find the TensorQuantizer and enable calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.enable_calib()\n...         module.disable_quant()  # Use full precision data to calibrate\n\n>>> # Feeding data samples\n>>> model(x)\n>>> # ...\n\n>>> # Finalize calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.load_calib_amax()\n...         module.enable_quant()\n\n>>> # If running on GPU, it needs to call .cuda() again because new tensors will be created by calibration process\n>>> model.cuda()\n\n>>> # Keep running the quantized model\n>>> # ...\n```\n\n### Export to ONNX\n\nThe goal of exporting to ONNX is to deploy inference by [TensorRT](https://developer.nvidia.com/tensorrt). Fake\nquantization will be broken into a pair of QuantizeLinear/DequantizeLinear ONNX ops."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 81c7152a-ae7b-419f-975c-44262b3336ef
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"81c7152a-ae7b-419f-975c-44262b3336ef\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84b1b074-3ecb-4005-8b6b-d264005c6912\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"65261afed7a170b012df35f61d1feaf43620c52efd42dbfb41212add7ba6d6ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"87d1aba4-44b7-4919-a84b-8c80c77b3be0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"99f384614fcbcb263691ae2639f11dcf8edf97c06de6d6fff330f06c72eb8c2e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0bf50155-5a83-46b4-9df1-6b55354b426b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"94d669c0582a3a62ad85cf27e03dbb584c20ba09374ff128c865a1a9938ac725\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3040, \"end_char_idx\": 4973, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "doc_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "ref_doc_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "chroma:document": "Example:\n\n```python\n>>> import pytorch_quantization.nn as quant_nn\n>>> from pytorch_quantization.tensor_quant import QuantDescriptor\n\n>>> # The default tensor quantizer is set to use Max calibration method\n>>> input_desc = QuantDescriptor(num_bits=8, calib_method=\"max\")\n>>> # The default tensor quantizer is set to be per-channel quantization for weights\n>>> weight_desc = QuantDescriptor(num_bits=8, axis=((0,)))\n>>> quant_nn.QuantLinear.set_default_quant_desc_input(input_desc)\n>>> quant_nn.QuantLinear.set_default_quant_desc_weight(weight_desc)\n```\n\n### Calibration\n\nCalibration is the terminology of passing data samples to the quantizer and deciding the best scaling factors for\ntensors. After setting up the tensor quantizers, one can use the following example to calibrate the model:\n\n```python\n>>> # Find the TensorQuantizer and enable calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.enable_calib()\n...         module.disable_quant()  # Use full precision data to calibrate\n\n>>> # Feeding data samples\n>>> model(x)\n>>> # ...\n\n>>> # Finalize calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.load_calib_amax()\n...         module.enable_quant()\n\n>>> # If running on GPU, it needs to call .cuda() again because new tensors will be created by calibration process\n>>> model.cuda()\n\n>>> # Keep running the quantized model\n>>> # ...\n```\n\n### Export to ONNX\n\nThe goal of exporting to ONNX is to deploy inference by [TensorRT](https://developer.nvidia.com/tensorrt). Fake\nquantization will be broken into a pair of QuantizeLinear/DequantizeLinear ONNX ops. After setting static member of\nTensorQuantizer to use Pytorch\u2019s own fake quantization functions, fake quantized model can be exported to ONNX, follow\nthe instructions in [torch.onnx](https://pytorch.org/docs/stable/onnx.html)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0bf50155-5a83-46b4-9df1-6b55354b426b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0bf50155-5a83-46b4-9df1-6b55354b426b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84b1b074-3ecb-4005-8b6b-d264005c6912\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"65261afed7a170b012df35f61d1feaf43620c52efd42dbfb41212add7ba6d6ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"81c7152a-ae7b-419f-975c-44262b3336ef\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fc766bfb1c3a6e18ee860722c5b28a8494004f47046681f543a8fe4c747edc19\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3734, \"end_char_idx\": 6292, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "doc_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "ref_doc_id": "84b1b074-3ecb-4005-8b6b-d264005c6912", "chroma:document": "After setting up the tensor quantizers, one can use the following example to calibrate the model:\n\n```python\n>>> # Find the TensorQuantizer and enable calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.enable_calib()\n...         module.disable_quant()  # Use full precision data to calibrate\n\n>>> # Feeding data samples\n>>> model(x)\n>>> # ...\n\n>>> # Finalize calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.load_calib_amax()\n...         module.enable_quant()\n\n>>> # If running on GPU, it needs to call .cuda() again because new tensors will be created by calibration process\n>>> model.cuda()\n\n>>> # Keep running the quantized model\n>>> # ...\n```\n\n### Export to ONNX\n\nThe goal of exporting to ONNX is to deploy inference by [TensorRT](https://developer.nvidia.com/tensorrt). Fake\nquantization will be broken into a pair of QuantizeLinear/DequantizeLinear ONNX ops. After setting static member of\nTensorQuantizer to use Pytorch\u2019s own fake quantization functions, fake quantized model can be exported to ONNX, follow\nthe instructions in [torch.onnx](https://pytorch.org/docs/stable/onnx.html). Example:\n\n```python\n>>> from pytorch_quantization.nn import TensorQuantizer\n\n>>> TensorQuantizer.use_fb_fake_quant = True\n\n>>> # Load the calibrated model\n>>> ...\n>>> # ONNX export\n>>> torch.onnx.export(...)\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## QDQBertConfig\n\n[[autodoc]] QDQBertConfig\n\n## QDQBertModel\n\n[[autodoc]] QDQBertModel\n    - forward\n\n## QDQBertLMHeadModel\n\n[[autodoc]] QDQBertLMHeadModel\n    - forward\n\n## QDQBertForMaskedLM\n\n[[autodoc]] QDQBertForMaskedLM\n    - forward\n\n## QDQBertForSequenceClassification\n\n[[autodoc]] QDQBertForSequenceClassification\n    - forward\n\n## QDQBertForNextSentencePrediction\n\n[[autodoc]] QDQBertForNextSentencePrediction\n    - forward\n\n## QDQBertForMultipleChoice\n\n[[autodoc]] QDQBertForMultipleChoice\n    - forward\n\n## QDQBertForTokenClassification\n\n[[autodoc]] QDQBertForTokenClassification\n    - forward\n\n## QDQBertForQuestionAnswering\n\n[[autodoc]] QDQBertForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0ffce875-02ed-4728-a436-6ffdf346cfea
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert-generation", "title": "BertGeneration", "tokens": 1045, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0ffce875-02ed-4728-a436-6ffdf346cfea\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f1810725-f8a1-453e-bd53-9d4983fb062b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e5912030493781a8d115b2d26521f3c4aca7cce9522c47363aed476cd04775e2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e5127b6f-93e1-4e63-a681-8d25af8afdf7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c58b9b3cc7d9f7c5e20fa00d0f57f2c6cb7d0fcfe6c7023bdd60d970576aaf52\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3380, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f1810725-f8a1-453e-bd53-9d4983fb062b", "doc_id": "f1810725-f8a1-453e-bd53-9d4983fb062b", "ref_doc_id": "f1810725-f8a1-453e-bd53-9d4983fb062b", "chroma:document": "# BertGeneration\n\n## Overview\n\nThe BertGeneration model is a BERT model that can be leveraged for sequence-to-sequence tasks using\n[`EncoderDecoderModel`] as proposed in [Leveraging Pre-trained Checkpoints for Sequence Generation\nTasks](https://arxiv.org/abs/1907.12461) by Sascha Rothe, Shashi Narayan, Aliaksei Severyn.\n\nThe abstract from the paper is the following:\n\n*Unsupervised pretraining of large neural models has recently revolutionized Natural Language Processing. By\nwarm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple\nbenchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language\nUnderstanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We\ndeveloped a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT,\nGPT-2 and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both\nencoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation,\nText Summarization, Sentence Splitting, and Sentence Fusion.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be\nfound [here](https://tfhub.dev/s?module-type=text-generation&subtype=module,placeholder).\n\n## Usage examples and tips\n\nThe model can be used in combination with the [`EncoderDecoderModel`] to leverage two pretrained BERT checkpoints for \nsubsequent fine-tuning:\n\n```python\n>>> # leverage checkpoints for Bert2Bert model...\n>>> # use BERT's cls token as BOS token and sep token as EOS token\n>>> encoder = BertGenerationEncoder.from_pretrained(\"google-bert/bert-large-uncased\", bos_token_id=101, eos_token_id=102)\n>>> # add cross attention layers and use BERT's cls token as BOS token and sep token as EOS token\n>>> decoder = BertGenerationDecoder.from_pretrained(\n...     \"google-bert/bert-large-uncased\", add_cross_attention=True, is_decoder=True, bos_token_id=101, eos_token_id=102\n... )\n>>> bert2bert = EncoderDecoderModel(encoder=encoder, decoder=decoder)\n\n>>> # create tokenizer...\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-large-uncased\")\n\n>>> input_ids = tokenizer(\n...     \"This is a long article to summarize\", add_special_tokens=False, return_tensors=\"pt\"\n... ).input_ids\n>>> labels = tokenizer(\"This is a short summary\", return_tensors=\"pt\").input_ids\n\n>>> # train...\n>>> loss = bert2bert(input_ids=input_ids, decoder_input_ids=labels, labels=labels).loss\n>>> loss.backward()\n```\n\nPretrained [`EncoderDecoderModel`] are also directly available in the model hub, e.g.:\n\n```python\n>>> # instantiate sentence fusion model\n>>> sentence_fuser = EncoderDecoderModel.from_pretrained(\"google/roberta2roberta_L-24_discofuse\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/roberta2roberta_L-24_discofuse\")\n\n>>> input_ids = tokenizer(\n...     \"This is the first sentence. This is the second sentence.\", add_special_tokens=False, return_tensors=\"pt\"\n... ).input_ids\n\n>>> outputs = sentence_fuser.generate(input_ids)\n\n>>> print(tokenizer.decode(outputs[0]))\n```\n\nTips:\n\n- [`BertGenerationEncoder`] and [`BertGenerationDecoder`] should be used in\n  combination with [`EncoderDecoder`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e5127b6f-93e1-4e63-a681-8d25af8afdf7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert-generation", "title": "BertGeneration", "tokens": 1045, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e5127b6f-93e1-4e63-a681-8d25af8afdf7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f1810725-f8a1-453e-bd53-9d4983fb062b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e5912030493781a8d115b2d26521f3c4aca7cce9522c47363aed476cd04775e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0ffce875-02ed-4728-a436-6ffdf346cfea\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2da781ba31e54516f3a267040f9fe115c51769f456baddb3bc032b8af4557c52\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2767, \"end_char_idx\": 3863, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f1810725-f8a1-453e-bd53-9d4983fb062b", "doc_id": "f1810725-f8a1-453e-bd53-9d4983fb062b", "ref_doc_id": "f1810725-f8a1-453e-bd53-9d4983fb062b", "chroma:document": ":\n\n```python\n>>> # instantiate sentence fusion model\n>>> sentence_fuser = EncoderDecoderModel.from_pretrained(\"google/roberta2roberta_L-24_discofuse\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/roberta2roberta_L-24_discofuse\")\n\n>>> input_ids = tokenizer(\n...     \"This is the first sentence. This is the second sentence.\", add_special_tokens=False, return_tensors=\"pt\"\n... ).input_ids\n\n>>> outputs = sentence_fuser.generate(input_ids)\n\n>>> print(tokenizer.decode(outputs[0]))\n```\n\nTips:\n\n- [`BertGenerationEncoder`] and [`BertGenerationDecoder`] should be used in\n  combination with [`EncoderDecoder`].\n- For summarization, sentence splitting, sentence fusion and translation, no special tokens are required for the input.\n  Therefore, no EOS token should be added to the end of the input.\n\n## BertGenerationConfig\n\n[[autodoc]] BertGenerationConfig\n\n## BertGenerationTokenizer\n\n[[autodoc]] BertGenerationTokenizer\n    - save_vocabulary\n\n## BertGenerationEncoder\n\n[[autodoc]] BertGenerationEncoder\n    - forward\n\n## BertGenerationDecoder\n\n[[autodoc]] BertGenerationDecoder\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0ff0bb27-8fd6-4102-994b-5715e2b19534
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0ff0bb27-8fd6-4102-994b-5715e2b19534\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c901ca29-9616-416a-8243-57061243bded\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37603d369f79b3b5422f08d31e4422131fe133ee5f4f569137e5e2aa7a9941f5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8f46116d-00ad-47bc-9275-cf7d25a83481\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a5607f6ecfb741e936af0da4910c5c2d5241e33ab33b18306371b5f0c6001b85\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2921, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c901ca29-9616-416a-8243-57061243bded", "doc_id": "c901ca29-9616-416a-8243-57061243bded", "ref_doc_id": "c901ca29-9616-416a-8243-57061243bded", "chroma:document": "# Transformer XL\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, so we won't accept any new PRs changing its code. This model was deprecated due to security issues linked to `pickle.load`.\n\nWe recommend switching to more recent models for improved security.\n\nIn case you would still like to use `TransfoXL` in your experiments, we recommend using the [Hub checkpoint](https://huggingface.co/transfo-xl/transfo-xl-wt103) with a specific revision to ensure you are downloading safe files from the Hub.\n\nYou will need to set the environment variable `TRUST_REMOTE_CODE` to `True` in order to allow the\nusage of `pickle.load()`:\n\n```python\nimport os\nfrom transformers import TransfoXLTokenizer, TransfoXLLMHeadModel\n\nos.environ[\"TRUST_REMOTE_CODE\"] = \"True\"\n\ncheckpoint = 'transfo-xl/transfo-xl-wt103'\nrevision = '40a186da79458c9f9de846edfaea79c412137f97'\n\ntokenizer = TransfoXLTokenizer.from_pretrained(checkpoint, revision=revision)\nmodel = TransfoXLLMHeadModel.from_pretrained(checkpoint, revision=revision)\n```\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.35.0.\nYou can do so by running the following command: `pip install -U transformers==4.35.0`.\n\n</Tip>\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=transfo-xl\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-transfo--xl-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/transfo-xl-wt103\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Transformer-XL model was proposed in [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan\nSalakhutdinov. It's a causal (uni-directional) transformer with relative positioning (sinuso\u00efdal) embeddings which can\nreuse previously computed hidden-states to attend to longer context (memory). This model also uses adaptive softmax\ninputs and outputs (tied).\n\nThe abstract from the paper is the following:\n\n*Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the\nsetting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency\nbeyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a\nnovel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the\ncontext fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450%\nlonger than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+\ntimes faster than vanilla Transformers during evaluation."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8f46116d-00ad-47bc-9275-cf7d25a83481
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8f46116d-00ad-47bc-9275-cf7d25a83481\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c901ca29-9616-416a-8243-57061243bded\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37603d369f79b3b5422f08d31e4422131fe133ee5f4f569137e5e2aa7a9941f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0ff0bb27-8fd6-4102-994b-5715e2b19534\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cd39914ad3513c09ce51bc1d102bbf13548c633af070abcefbf0cb67ff424b39\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"40fad455-f76b-42ad-be4e-1421d1149b00\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c7c7442136b16edd0ce74dbb3c2745722cc5cfe04297b008c536f202bf9950e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1868, \"end_char_idx\": 4701, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c901ca29-9616-416a-8243-57061243bded", "doc_id": "c901ca29-9616-416a-8243-57061243bded", "ref_doc_id": "c901ca29-9616-416a-8243-57061243bded", "chroma:document": "It's a causal (uni-directional) transformer with relative positioning (sinuso\u00efdal) embeddings which can\nreuse previously computed hidden-states to attend to longer context (memory). This model also uses adaptive softmax\ninputs and outputs (tied).\n\nThe abstract from the paper is the following:\n\n*Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the\nsetting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency\nbeyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a\nnovel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the\ncontext fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450%\nlonger than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+\ntimes faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of\nbpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn\nTreebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably\ncoherent, novel text articles with thousands of tokens.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/kimiyoung/transformer-xl).\n\n## Usage tips\n\n- Transformer-XL uses relative sinusoidal positional embeddings. Padding can be done on the left or on the right. The\n  original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left.\n- Transformer-XL is one of the few models that has no sequence length limit.\n- Same as a regular GPT model, but introduces a recurrence mechanism for two consecutive segments (similar to a regular RNNs with two consecutive inputs). In this context, a segment is a number of consecutive tokens (for instance 512) that may span across multiple documents, and segments are fed in order to the model.\n- Basically, the hidden states of the previous segment are concatenated to the current input to compute the attention scores. This allows the model to pay attention to information that was in the previous segment as well as the current one. By stacking multiple attention layers, the receptive field can be increased to multiple previous segments.\n- This changes the positional embeddings to positional relative embeddings (as the regular positional embeddings would give the same results in the current input and the current hidden state at a given position) and needs to make some adjustments in the way attention scores are computed."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 40fad455-f76b-42ad-be4e-1421d1149b00
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"40fad455-f76b-42ad-be4e-1421d1149b00\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c901ca29-9616-416a-8243-57061243bded\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37603d369f79b3b5422f08d31e4422131fe133ee5f4f569137e5e2aa7a9941f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8f46116d-00ad-47bc-9275-cf7d25a83481\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e2251252f3db8b85374e3dff9964de424ecf8a7d96e209aa41254ed8a088bb6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2922, \"end_char_idx\": 6171, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c901ca29-9616-416a-8243-57061243bded", "doc_id": "c901ca29-9616-416a-8243-57061243bded", "ref_doc_id": "c901ca29-9616-416a-8243-57061243bded", "chroma:document": "Notably, we improve the state-of-the-art results of\nbpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn\nTreebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably\ncoherent, novel text articles with thousands of tokens.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/kimiyoung/transformer-xl).\n\n## Usage tips\n\n- Transformer-XL uses relative sinusoidal positional embeddings. Padding can be done on the left or on the right. The\n  original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left.\n- Transformer-XL is one of the few models that has no sequence length limit.\n- Same as a regular GPT model, but introduces a recurrence mechanism for two consecutive segments (similar to a regular RNNs with two consecutive inputs). In this context, a segment is a number of consecutive tokens (for instance 512) that may span across multiple documents, and segments are fed in order to the model.\n- Basically, the hidden states of the previous segment are concatenated to the current input to compute the attention scores. This allows the model to pay attention to information that was in the previous segment as well as the current one. By stacking multiple attention layers, the receptive field can be increased to multiple previous segments.\n- This changes the positional embeddings to positional relative embeddings (as the regular positional embeddings would give the same results in the current input and the current hidden state at a given position) and needs to make some adjustments in the way attention scores are computed.\n\n\n<Tip warning={true}>\n\nTransformerXL does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035)\n\n</Tip>\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## TransfoXLConfig\n\n[[autodoc]] TransfoXLConfig\n\n## TransfoXLTokenizer\n\n[[autodoc]] TransfoXLTokenizer\n    - save_vocabulary\n\n## TransfoXL specific outputs\n\n[[autodoc]] models.deprecated.transfo_xl.modeling_transfo_xl.TransfoXLModelOutput\n\n[[autodoc]] models.deprecated.transfo_xl.modeling_transfo_xl.TransfoXLLMHeadModelOutput\n\n[[autodoc]] models.deprecated.transfo_xl.modeling_tf_transfo_xl.TFTransfoXLModelOutput\n\n[[autodoc]] models.deprecated.transfo_xl.modeling_tf_transfo_xl.TFTransfoXLLMHeadModelOutput\n\n<frameworkcontent>\n<pt>\n\n## TransfoXLModel\n\n[[autodoc]] TransfoXLModel\n    - forward\n\n## TransfoXLLMHeadModel\n\n[[autodoc]] TransfoXLLMHeadModel\n    - forward\n\n## TransfoXLForSequenceClassification\n\n[[autodoc]] TransfoXLForSequenceClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFTransfoXLModel\n\n[[autodoc]] TFTransfoXLModel\n    - call\n\n## TFTransfoXLLMHeadModel\n\n[[autodoc]] TFTransfoXLLMHeadModel\n    - call\n\n## TFTransfoXLForSequenceClassification\n\n[[autodoc]] TFTransfoXLForSequenceClassification\n    - call\n\n</tf>\n</frameworkcontent>\n\n## Internal Layers\n\n[[autodoc]] AdaptiveEmbedding\n\n[[autodoc]] TFAdaptiveEmbedding"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e9727976-bb04-4173-b01c-8444d4232642
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deta", "title": "DETA", "tokens": 1005, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e9727976-bb04-4173-b01c-8444d4232642\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8595986a-2a44-4b3c-854b-0165ce1793d1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a2b9adf121c0c5de2026423a9e4cddd169b0e1e0d3f34dbc4ede6b87afe32a16\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50697310-3e47-4456-8c6d-64e5b6c3eacf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"942e832408de5b9f8a5d478a15063d8e2fa0c770fa1399e90031e39bcf7a1b34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3403, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8595986a-2a44-4b3c-854b-0165ce1793d1", "doc_id": "8595986a-2a44-4b3c-854b-0165ce1793d1", "ref_doc_id": "8595986a-2a44-4b3c-854b-0165ce1793d1", "chroma:document": "# DETA\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe DETA model was proposed in [NMS Strikes Back](https://arxiv.org/abs/2212.06137) by Jeffrey Ouyang-Zhang, Jang Hyun Cho, Xingyi Zhou, Philipp Kr\u00e4henb\u00fchl.\nDETA (short for Detection Transformers with Assignment) improves [Deformable DETR](deformable_detr) by replacing the one-to-one bipartite Hungarian matching loss\nwith one-to-many label assignments used in traditional detectors with non-maximum suppression (NMS). This leads to significant gains of up to 2.5 mAP.\n\nThe abstract from the paper is the following:\n\n*Detection Transformer (DETR) directly transforms queries to unique objects by using one-to-one bipartite matching during training and enables end-to-end object detection. Recently, these models have surpassed traditional detectors on COCO with undeniable elegance. However, they differ from traditional detectors in multiple designs, including model architecture and training schedules, and thus the effectiveness of one-to-one matching is not fully understood. In this work, we conduct a strict comparison between the one-to-one Hungarian matching in DETRs and the one-to-many label assignments in traditional detectors with non-maximum supervision (NMS). Surprisingly, we observe one-to-many assignments with NMS consistently outperform standard one-to-one matching under the same setting, with a significant gain of up to 2.5 mAP. Our detector that trains Deformable-DETR with traditional IoU-based label assignment achieved 50.2 COCO mAP within 12 epochs (1x schedule) with ResNet50 backbone, outperforming all existing traditional or transformer-based detectors in this setting. On multiple datasets, schedules, and architectures, we consistently show bipartite matching is unnecessary for performant detection transformers. Furthermore, we attribute the success of detection transformers to their expressive transformer architecture.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/deta_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> DETA overview. Taken from the <a href=\"https://arxiv.org/abs/2212.06137\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/jozhang97/DETA).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DETA.\n\n- Demo notebooks for DETA can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETA).\n- Scripts for finetuning [`DetaForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 50697310-3e47-4456-8c6d-64e5b6c3eacf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deta", "title": "DETA", "tokens": 1005, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50697310-3e47-4456-8c6d-64e5b6c3eacf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8595986a-2a44-4b3c-854b-0165ce1793d1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a2b9adf121c0c5de2026423a9e4cddd169b0e1e0d3f34dbc4ede6b87afe32a16\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e9727976-bb04-4173-b01c-8444d4232642\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6cbe8139ba07cc7290232551a0a9c1c27c765ab1eb1410d11473d26fded2fc55\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1696, \"end_char_idx\": 3678, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8595986a-2a44-4b3c-854b-0165ce1793d1", "doc_id": "8595986a-2a44-4b3c-854b-0165ce1793d1", "ref_doc_id": "8595986a-2a44-4b3c-854b-0165ce1793d1", "chroma:document": "Our detector that trains Deformable-DETR with traditional IoU-based label assignment achieved 50.2 COCO mAP within 12 epochs (1x schedule) with ResNet50 backbone, outperforming all existing traditional or transformer-based detectors in this setting. On multiple datasets, schedules, and architectures, we consistently show bipartite matching is unnecessary for performant detection transformers. Furthermore, we attribute the success of detection transformers to their expressive transformer architecture.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/deta_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> DETA overview. Taken from the <a href=\"https://arxiv.org/abs/2212.06137\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/jozhang97/DETA).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DETA.\n\n- Demo notebooks for DETA can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETA).\n- Scripts for finetuning [`DetaForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DetaConfig\n\n[[autodoc]] DetaConfig\n\n## DetaImageProcessor\n\n[[autodoc]] DetaImageProcessor\n    - preprocess\n    - post_process_object_detection\n\n## DetaModel\n\n[[autodoc]] DetaModel\n    - forward\n\n## DetaForObjectDetection\n\n[[autodoc]] DetaForObjectDetection\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d64930a0-ce59-420a-92eb-13249950685a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/starcoder2", "title": "Starcoder2", "tokens": 1389, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d64930a0-ce59-420a-92eb-13249950685a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6a5d0cf6-aa93-4d9d-8282-aef9141f287b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"322ac5d3ca3bc4886f79586ea22e7ab6dd953deb92aa0e921ce4a3fe330bcdc7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b0a6c9d2-579e-40b6-b645-5a1f9ae1cdac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a64dc0b39a2a57ffb34c1d76c9296ba482aa2d91a2cabfa9c201ff13861ee2c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2653, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6a5d0cf6-aa93-4d9d-8282-aef9141f287b", "doc_id": "6a5d0cf6-aa93-4d9d-8282-aef9141f287b", "ref_doc_id": "6a5d0cf6-aa93-4d9d-8282-aef9141f287b", "chroma:document": "# Starcoder2\n\n## Overview\n\nStarCoder2 is a family of open LLMs for code and comes in 3 different sizes with 3B, 7B and 15B parameters. The flagship StarCoder2-15B model is trained on over 4 trillion tokens and 600+ programming languages from The Stack v2. All models use Grouped Query Attention, a context window of 16,384 tokens with a sliding window attention of 4,096 tokens, and were trained using the Fill-in-the-Middle objective. The models have been released with the paper [StarCoder 2 and The Stack v2: The Next Generation](https://arxiv.org/abs/2402.19173) by Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krau\u00df, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Mu\u00f1oz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.\n\nThe abstract of the paper is the following:\n\n> The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b0a6c9d2-579e-40b6-b645-5a1f9ae1cdac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/starcoder2", "title": "Starcoder2", "tokens": 1389, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b0a6c9d2-579e-40b6-b645-5a1f9ae1cdac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6a5d0cf6-aa93-4d9d-8282-aef9141f287b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"322ac5d3ca3bc4886f79586ea22e7ab6dd953deb92aa0e921ce4a3fe330bcdc7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d64930a0-ce59-420a-92eb-13249950685a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f38e44d51405b6e58ee9b10289654730935b4d5af46b71644678852518326931\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1547, \"end_char_idx\": 4578, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6a5d0cf6-aa93-4d9d-8282-aef9141f287b", "doc_id": "6a5d0cf6-aa93-4d9d-8282-aef9141f287b", "ref_doc_id": "6a5d0cf6-aa93-4d9d-8282-aef9141f287b", "chroma:document": "The abstract of the paper is the following:\n\n> The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size. Although DeepSeekCoder- 33B is the best-performing model at code completion for high-resource languages, we find that StarCoder2-15B outperforms it on math and code reasoning benchmarks, as well as several low-resource languages. We make the model weights available under an OpenRAIL license and ensure full transparency regarding the training data by releasing the SoftWare Heritage persistent IDentifiers (SWHIDs) of the source code data.\n## License\n\nThe models are licensed under the [BigCode OpenRAIL-M v1 license agreement](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement).\n \n## Usage tips\n\nThe StarCoder2 models can be found in the [HuggingFace hub](https://huggingface.co/collections/bigcode/starcoder2-65de6da6e87db3383572be1a). You can find some examples for inference and fine-tuning in StarCoder2's [GitHub repo](https://github.com/bigcode-project/starcoder2).\n\nThese ready-to-use checkpoints can be downloaded and used via the HuggingFace Hub:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"bigcode/starcoder2-7b\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"bigcode/starcoder2-7b\")\n\n>>> prompt = \"def print_hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=10, do_sample=False)\n>>> tokenizer.batch_decode(generated_ids)[0]\n'def print_hello_world():\\n    print(\"Hello World!\")\\n\\ndef print'\n```\n\n## Starcoder2Config\n\n[[autodoc]] Starcoder2Config\n\n## Starcoder2Model\n\n[[autodoc]] Starcoder2Model\n    - forward\n\n## Starcoder2ForCausalLM\n\n[[autodoc]] Starcoder2ForCausalLM\n    - forward\n\n## Starcoder2ForSequenceClassification\n\n[[autodoc]] Starcoder2ForSequenceClassification\n    - forward\n\n## Starcoder2ForTokenClassification\n\n[[autodoc]] Starcoder2ForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e4b3ad27-6de7-43bc-9c21-8a9d371ba9cd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/electra", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e4b3ad27-6de7-43bc-9c21-8a9d371ba9cd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3fe54bad-d9cf-41cb-9c63-5d41b9a58faa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"645e69e7481296f4dd5c8d540d7817982308c4722a60d7f3029a120d7a318c24\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8666e6c6-e29e-4eaa-af66-d53d714e36ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ee65d2f4c8537c448c77e9c2366a62cb1168d338e2ed0bb0d815c8f26d30eb0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3307, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "doc_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "ref_doc_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "chroma:document": "# ELECTRA\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=electra\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-electra-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/electra_large_discriminator_squad2_512\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe ELECTRA model was proposed in the paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than\nGenerators](https://openreview.net/pdf?id=r1xMH1BtvB). ELECTRA is a new pretraining approach which trains two\ntransformer models: the generator and the discriminator. The generator's role is to replace tokens in a sequence, and\nis therefore trained as a masked language model. The discriminator, which is the model we're interested in, tries to\nidentify which tokens were replaced by the generator in the sequence.\n\nThe abstract from the paper is the following:\n\n*Masked language modeling (MLM) pretraining methods such as BERT corrupt the input by replacing some tokens with [MASK]\nand then train a model to reconstruct the original tokens. While they produce good results when transferred to\ndownstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a\nmore sample-efficient pretraining task called replaced token detection. Instead of masking the input, our approach\ncorrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead\nof training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that\npredicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments\ndemonstrate this new pretraining task is more efficient than MLM because the task is defined over all input tokens\nrather than just the small subset that was masked out. As a result, the contextual representations learned by our\napproach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are\nparticularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained\nusing 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale,\nwhere it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when\nusing the same amount of compute.*\n\nThis model was contributed by [lysandre](https://huggingface.co/lysandre). The original code can be found [here](https://github.com/google-research/electra).\n\n## Usage tips\n\n- ELECTRA is the pretraining approach, therefore there is nearly no changes done to the underlying model: BERT. The\n  only change is the separation of the embedding size and the hidden size: the embedding size is generally smaller,\n  while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from their\n  embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no projection\n  layer is used.\n- ELECTRA is a transformer model pretrained with the use of another (small) masked language model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8666e6c6-e29e-4eaa-af66-d53d714e36ad
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/electra", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8666e6c6-e29e-4eaa-af66-d53d714e36ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3fe54bad-d9cf-41cb-9c63-5d41b9a58faa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"645e69e7481296f4dd5c8d540d7817982308c4722a60d7f3029a120d7a318c24\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4b3ad27-6de7-43bc-9c21-8a9d371ba9cd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"56ed36abc747083ae4813924a63473af854bcabc57a049332365e664d3d6c31c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1f5a292c-632e-45c1-a6f2-a1fb193225e4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d56f0afe7b1d566508da7dc2f7bd94833395dcd218b5aaeadc147c935a3e9060\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "doc_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "ref_doc_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "chroma:document": "Then, instead\nof training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that\npredicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments\ndemonstrate this new pretraining task is more efficient than MLM because the task is defined over all input tokens\nrather than just the small subset that was masked out. As a result, the contextual representations learned by our\napproach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are\nparticularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained\nusing 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale,\nwhere it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when\nusing the same amount of compute.*\n\nThis model was contributed by [lysandre](https://huggingface.co/lysandre). The original code can be found [here](https://github.com/google-research/electra).\n\n## Usage tips\n\n- ELECTRA is the pretraining approach, therefore there is nearly no changes done to the underlying model: BERT. The\n  only change is the separation of the embedding size and the hidden size: the embedding size is generally smaller,\n  while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from their\n  embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no projection\n  layer is used.\n- ELECTRA is a transformer model pretrained with the use of another (small) masked language model. The inputs are corrupted by that language model, which takes an input text that is randomly masked and outputs a text in which ELECTRA has to predict which token is an original and which one has been replaced. Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps.\n- The ELECTRA checkpoints saved using [Google Research's implementation](https://github.com/google-research/electra)\n  contain both the generator and discriminator. The conversion script requires the user to name which model to export\n  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all\n  available ELECTRA models, however. This means that the discriminator may be loaded in the\n  [`ElectraForMaskedLM`] model, and the generator may be loaded in the\n  [`ElectraForPreTraining`] model (the classification head will be randomly initialized as it\n  doesn't exist in the generator).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## ElectraConfig\n\n[[autodoc]] ElectraConfig\n\n## ElectraTokenizer\n\n[[autodoc]] ElectraTokenizer\n\n## ElectraTokenizerFast\n\n[[autodoc]] ElectraTokenizerFast\n\n## Electra specific outputs\n\n[[autodoc]] models.electra.modeling_electra."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1f5a292c-632e-45c1-a6f2-a1fb193225e4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/electra", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1f5a292c-632e-45c1-a6f2-a1fb193225e4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3fe54bad-d9cf-41cb-9c63-5d41b9a58faa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"645e69e7481296f4dd5c8d540d7817982308c4722a60d7f3029a120d7a318c24\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8666e6c6-e29e-4eaa-af66-d53d714e36ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6e161dafb893ba9d3d684570ecd4205c482770024f9ac0329c7518138eee7197\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a9143402-7f4a-4712-94eb-2cebede72084\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cf452d2274695614f192213ebfad351cbdf3a420e255c6d895d3e0e165190b1f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "doc_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "ref_doc_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "chroma:document": "The inputs are corrupted by that language model, which takes an input text that is randomly masked and outputs a text in which ELECTRA has to predict which token is an original and which one has been replaced. Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps.\n- The ELECTRA checkpoints saved using [Google Research's implementation](https://github.com/google-research/electra)\n  contain both the generator and discriminator. The conversion script requires the user to name which model to export\n  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all\n  available ELECTRA models, however. This means that the discriminator may be loaded in the\n  [`ElectraForMaskedLM`] model, and the generator may be loaded in the\n  [`ElectraForPreTraining`] model (the classification head will be randomly initialized as it\n  doesn't exist in the generator).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## ElectraConfig\n\n[[autodoc]] ElectraConfig\n\n## ElectraTokenizer\n\n[[autodoc]] ElectraTokenizer\n\n## ElectraTokenizerFast\n\n[[autodoc]] ElectraTokenizerFast\n\n## Electra specific outputs\n\n[[autodoc]] models.electra.modeling_electra.ElectraForPreTrainingOutput\n\n[[autodoc]] models.electra.modeling_tf_electra."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a9143402-7f4a-4712-94eb-2cebede72084
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/electra", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a9143402-7f4a-4712-94eb-2cebede72084\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3fe54bad-d9cf-41cb-9c63-5d41b9a58faa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"645e69e7481296f4dd5c8d540d7817982308c4722a60d7f3029a120d7a318c24\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1f5a292c-632e-45c1-a6f2-a1fb193225e4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9453577a9abf786d0c4f04eb74837520fb262a0a6bac450e2dc270e1ea2571cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "doc_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "ref_doc_id": "3fe54bad-d9cf-41cb-9c63-5d41b9a58faa", "chroma:document": "Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps.\n- The ELECTRA checkpoints saved using [Google Research's implementation](https://github.com/google-research/electra)\n  contain both the generator and discriminator. The conversion script requires the user to name which model to export\n  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all\n  available ELECTRA models, however. This means that the discriminator may be loaded in the\n  [`ElectraForMaskedLM`] model, and the generator may be loaded in the\n  [`ElectraForPreTraining`] model (the classification head will be randomly initialized as it\n  doesn't exist in the generator).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## ElectraConfig\n\n[[autodoc]] ElectraConfig\n\n## ElectraTokenizer\n\n[[autodoc]] ElectraTokenizer\n\n## ElectraTokenizerFast\n\n[[autodoc]] ElectraTokenizerFast\n\n## Electra specific outputs\n\n[[autodoc]] models.electra.modeling_electra.ElectraForPreTrainingOutput\n\n[[autodoc]] models.electra.modeling_tf_electra.TFElectraForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## ElectraModel\n\n[[autodoc]] ElectraModel\n    - forward\n\n## ElectraForPreTraining\n\n[[autodoc]] ElectraForPreTraining\n    - forward\n\n## ElectraForCausalLM\n\n[[autodoc]] ElectraForCausalLM\n    - forward\n\n## ElectraForMaskedLM\n\n[[autodoc]] ElectraForMaskedLM\n    - forward\n\n## ElectraForSequenceClassification\n\n[[autodoc]] ElectraForSequenceClassification\n    - forward\n\n## ElectraForMultipleChoice\n\n[[autodoc]] ElectraForMultipleChoice\n    - forward\n\n## ElectraForTokenClassification\n\n[[autodoc]] ElectraForTokenClassification\n    - forward\n\n## ElectraForQuestionAnswering\n\n[[autodoc]] ElectraForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFElectraModel\n\n[[autodoc]] TFElectraModel\n    - call\n\n## TFElectraForPreTraining\n\n[[autodoc]] TFElectraForPreTraining\n    - call\n\n## TFElectraForMaskedLM\n\n[[autodoc]] TFElectraForMaskedLM\n    - call\n\n## TFElectraForSequenceClassification\n\n[[autodoc]] TFElectraForSequenceClassification\n    - call\n\n## TFElectraForMultipleChoice\n\n[[autodoc]] TFElectraForMultipleChoice\n    - call\n\n## TFElectraForTokenClassification\n\n[[autodoc]] TFElectraForTokenClassification\n    - call\n\n## TFElectraForQuestionAnswering\n\n[[autodoc]] TFElectraForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxElectraModel\n\n[[autodoc]] FlaxElectraModel\n    - __call__\n\n## FlaxElectraForPreTraining\n\n[[autodoc]] FlaxElectraForPreTraining\n    - __call__\n\n## FlaxElectraForCausalLM\n\n[[autodoc]] FlaxElectraForCausalLM\n    - __call__\n\n## FlaxElectraForMaskedLM\n\n[[autodoc]] FlaxElectraForMaskedLM\n    - __call__\n\n## FlaxElectraForSequenceClassification\n\n[[autodoc]] FlaxElectraForSequenceClassification\n    - __call__\n\n## FlaxElectraForMultipleChoice\n\n[[autodoc]] FlaxElectraForMultipleChoice\n    - __call__\n\n## FlaxElectraForTokenClassification\n\n[[autodoc]] FlaxElectraForTokenClassification\n    - __call__\n\n## FlaxElectraForQuestionAnswering\n\n[[autodoc]] FlaxElectraForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f7c9d297-34cd-4e92-83af-81ff6096d562
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm", "title": "RoBERTa-PreLayerNorm", "tokens": 1289, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f7c9d297-34cd-4e92-83af-81ff6096d562\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ae5d6d98-6baa-4348-9759-1b4a34c0ca68\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"256b38168415c942b2c3d38c14215f02d6881e6e0c499ac3b301319b78a66d34\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3b6b0547-c172-48a3-8abd-ed8b4b7541af\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"34ec90fcec75459229a09d392332aa98ceab3b1400c7f1bf6268f54baa0d3797\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ae5d6d98-6baa-4348-9759-1b4a34c0ca68", "doc_id": "ae5d6d98-6baa-4348-9759-1b4a34c0ca68", "ref_doc_id": "ae5d6d98-6baa-4348-9759-1b4a34c0ca68", "chroma:document": "# RoBERTa-PreLayerNorm\n\n## Overview\n\nThe RoBERTa-PreLayerNorm model was proposed in [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://arxiv.org/abs/1904.01038) by Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli.\nIt is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).\n\nThe abstract from the paper is the following:\n\n*fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs.*\n\nThis model was contributed by [andreasmaden](https://huggingface.co/andreasmadsen).\nThe original code can be found [here](https://github.com/princeton-nlp/DinkyTrain).\n\n## Usage tips\n\n- The implementation is the same as [Roberta](roberta) except instead of using _Add and Norm_ it does _Norm and Add_. _Add_ and _Norm_ refers to the Addition and LayerNormalization as described in [Attention Is All You Need](https://arxiv.org/abs/1706.03762).\n- This is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3b6b0547-c172-48a3-8abd-ed8b4b7541af
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm", "title": "RoBERTa-PreLayerNorm", "tokens": 1289, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3b6b0547-c172-48a3-8abd-ed8b4b7541af\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ae5d6d98-6baa-4348-9759-1b4a34c0ca68\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"256b38168415c942b2c3d38c14215f02d6881e6e0c499ac3b301319b78a66d34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f7c9d297-34cd-4e92-83af-81ff6096d562\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c9dce239eed6f06b0006564d8c5eff2a3b6fe5610756900b5d76725b1257d28c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ae5d6d98-6baa-4348-9759-1b4a34c0ca68", "doc_id": "ae5d6d98-6baa-4348-9759-1b4a34c0ca68", "ref_doc_id": "ae5d6d98-6baa-4348-9759-1b4a34c0ca68", "chroma:document": "It is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).\n\nThe abstract from the paper is the following:\n\n*fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs.*\n\nThis model was contributed by [andreasmaden](https://huggingface.co/andreasmadsen).\nThe original code can be found [here](https://github.com/princeton-nlp/DinkyTrain).\n\n## Usage tips\n\n- The implementation is the same as [Roberta](roberta) except instead of using _Add and Norm_ it does _Norm and Add_. _Add_ and _Norm_ refers to the Addition and LayerNormalization as described in [Attention Is All You Need](https://arxiv.org/abs/1706.03762).\n- This is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## RobertaPreLayerNormConfig\n\n[[autodoc]] RobertaPreLayerNormConfig\n\n<frameworkcontent>\n<pt>\n\n## RobertaPreLayerNormModel\n\n[[autodoc]] RobertaPreLayerNormModel\n    - forward\n\n## RobertaPreLayerNormForCausalLM\n\n[[autodoc]] RobertaPreLayerNormForCausalLM\n    - forward\n\n## RobertaPreLayerNormForMaskedLM\n\n[[autodoc]] RobertaPreLayerNormForMaskedLM\n    - forward\n\n## RobertaPreLayerNormForSequenceClassification\n\n[[autodoc]] RobertaPreLayerNormForSequenceClassification\n    - forward\n\n## RobertaPreLayerNormForMultipleChoice\n\n[[autodoc]] RobertaPreLayerNormForMultipleChoice\n    - forward\n\n## RobertaPreLayerNormForTokenClassification\n\n[[autodoc]] RobertaPreLayerNormForTokenClassification\n    - forward\n\n## RobertaPreLayerNormForQuestionAnswering\n\n[[autodoc]] RobertaPreLayerNormForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFRobertaPreLayerNormModel\n\n[[autodoc]] TFRobertaPreLayerNormModel\n    - call\n\n## TFRobertaPreLayerNormForCausalLM\n\n[[autodoc]] TFRobertaPreLayerNormForCausalLM\n    - call\n\n## TFRobertaPreLayerNormForMaskedLM\n\n[[autodoc]] TFRobertaPreLayerNormForMaskedLM\n    - call\n\n## TFRobertaPreLayerNormForSequenceClassification\n\n[[autodoc]] TFRobertaPreLayerNormForSequenceClassification\n    - call\n\n## TFRobertaPreLayerNormForMultipleChoice\n\n[[autodoc]] TFRobertaPreLayerNormForMultipleChoice\n    - call\n\n## TFRobertaPreLayerNormForTokenClassification\n\n[[autodoc]] TFRobertaPreLayerNormForTokenClassification\n    - call\n\n## TFRobertaPreLayerNormForQuestionAnswering\n\n[[autodoc]] TFRobertaPreLayerNormForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxRobertaPreLayerNormModel\n\n[[autodoc]] FlaxRobertaPreLayerNormModel\n    - __call__\n\n## FlaxRobertaPreLayerNormForCausalLM\n\n[[autodoc]] FlaxRobertaPreLayerNormForCausalLM\n    - __call__\n\n## FlaxRobertaPreLayerNormForMaskedLM\n\n[[autodoc]] FlaxRobertaPreLayerNormForMaskedLM\n    - __call__\n\n## FlaxRobertaPreLayerNormForSequenceClassification\n\n[[autodoc]] FlaxRobertaPreLayerNormForSequenceClassification\n    - __call__\n\n## FlaxRobertaPreLayerNormForMultipleChoice\n\n[[autodoc]] FlaxRobertaPreLayerNormForMultipleChoice\n    - __call__\n\n## FlaxRobertaPreLayerNormForTokenClassification\n\n[[autodoc]] FlaxRobertaPreLayerNormForTokenClassification\n    - __call__\n\n## FlaxRobertaPreLayerNormForQuestionAnswering\n\n[[autodoc]] FlaxRobertaPreLayerNormForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 840a95ea-73e3-4372-8748-8ebe05cb1ba2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilevitv2", "title": "MobileViTV2", "tokens": 862, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"840a95ea-73e3-4372-8748-8ebe05cb1ba2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevitv2\", \"title\": \"MobileViTV2\", \"tokens\": 862, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8cf57301-7883-42e5-969a-5877bd470026\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevitv2\", \"title\": \"MobileViTV2\", \"tokens\": 862, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"58986bbf410fa2f3c09a0ee5ec7a18e8a726c84ed6f5a1723452952b69cfd6de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2960, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8cf57301-7883-42e5-969a-5877bd470026", "doc_id": "8cf57301-7883-42e5-969a-5877bd470026", "ref_doc_id": "8cf57301-7883-42e5-969a-5877bd470026", "chroma:document": "# MobileViTV2\n\n## Overview\n\nThe MobileViTV2 model was proposed in [Separable Self-attention for Mobile Vision Transformers](https://arxiv.org/abs/2206.02680) by Sachin Mehta and Mohammad Rastegari.\n\nMobileViTV2 is the second version of MobileViT, constructed by replacing the multi-headed self-attention in MobileViT with separable self-attention.\n\nThe abstract from the paper is the following:\n\n*Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network-based models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires O(k2) time complexity with respect to the number of tokens (or patches) k. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrained devices. This paper introduces a separable self-attention method with linear complexity, i.e. O(k). A simple yet effective characteristic of the proposed method is that it uses element-wise operations for computing self-attention, making it a good choice for resource-constrained devices. The improved model, MobileViTV2, is state-of-the-art on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTV2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileViT by about 1% while running 3.2\u00d7 faster on a mobile device.*\n\nThis model was contributed by [shehan97](https://huggingface.co/shehan97).\nThe original code can be found [here](https://github.com/apple/ml-cvnets).\n\n## Usage tips\n\n- MobileViTV2 is more like a CNN than a Transformer model. It does not work on sequence data but on batches of images. Unlike ViT, there are no embeddings. The backbone model outputs a feature map.\n- One can use [`MobileViTImageProcessor`] to prepare images for the model. Note that if you do your own preprocessing, the pretrained checkpoints expect images to be in BGR pixel order (not RGB).\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes).\n- The segmentation model uses a [DeepLabV3](https://arxiv.org/abs/1706.05587) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).\n\n## MobileViTV2Config\n\n[[autodoc]] MobileViTV2Config\n\n## MobileViTV2Model\n\n[[autodoc]] MobileViTV2Model\n    - forward\n\n## MobileViTV2ForImageClassification\n\n[[autodoc]] MobileViTV2ForImageClassification\n    - forward\n\n## MobileViTV2ForSemanticSegmentation\n\n[[autodoc]] MobileViTV2ForSemanticSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cb35cd39-8eb9-4757-ad6f-91e54efacea8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cb35cd39-8eb9-4757-ad6f-91e54efacea8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c492a720-2872-4241-aa10-3bfc474c2143\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2eb4d165131951bb06541e07ce3f2b388d9f65af14dc2bae06ed20b902632072\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7cbd9cc1-749b-4351-b2d1-8522d58f3334\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"188b9d991439a2fed573f6a51c522ea5cc660eef77e7d94250fb2192ff3b67c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2725, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c492a720-2872-4241-aa10-3bfc474c2143", "doc_id": "c492a720-2872-4241-aa10-3bfc474c2143", "ref_doc_id": "c492a720-2872-4241-aa10-3bfc474c2143", "chroma:document": "# Grounding DINO\n\n## Overview\n\nThe Grounding DINO model was proposed in [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://arxiv.org/abs/2303.05499) by Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang. Grounding DINO extends a closed-set object detection model with a text encoder, enabling open-set object detection. The model achieves remarkable results, such as 52.5 AP on COCO zero-shot.\n\nThe abstract from the paper is the following:\n\n*In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a 52.5 AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean 26.1 AP.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grouding_dino_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Grounding DINO overview. Taken from the <a href=\"https://arxiv.org/abs/2303.05499\">original paper</a>. </small>\n\nThis model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/IDEA-Research/GroundingDINO).\n\n## Usage tips\n\n- One can use [`GroundingDinoProcessor`] to prepare image-text pairs for the model.\n- To separate classes in the text use a period e.g. \"a cat. a dog.\"\n- When using multiple classes (e.g. `\"a cat. a dog.\"`), use `post_process_grounded_object_detection` from [`GroundingDinoProcessor`] to post process outputs. Since, the labels returned from `post_process_object_detection` represent the indices from the model dimension where prob > threshold."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7cbd9cc1-749b-4351-b2d1-8522d58f3334
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7cbd9cc1-749b-4351-b2d1-8522d58f3334\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c492a720-2872-4241-aa10-3bfc474c2143\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2eb4d165131951bb06541e07ce3f2b388d9f65af14dc2bae06ed20b902632072\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb35cd39-8eb9-4757-ad6f-91e54efacea8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"814592e1bde35fd72b98d61615dcef7a70667bfa41a648154a12a1dc0e3f86b1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"46c770f1-7e97-4bac-90e0-e26f71ef7884\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fef42d10ad6741160fdb6ec00f22268db4231f8acca715345a7600180a4b044d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1225, \"end_char_idx\": 3373, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c492a720-2872-4241-aa10-3bfc474c2143", "doc_id": "c492a720-2872-4241-aa10-3bfc474c2143", "ref_doc_id": "c492a720-2872-4241-aa10-3bfc474c2143", "chroma:document": "While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a 52.5 AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean 26.1 AP.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grouding_dino_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Grounding DINO overview. Taken from the <a href=\"https://arxiv.org/abs/2303.05499\">original paper</a>. </small>\n\nThis model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/IDEA-Research/GroundingDINO).\n\n## Usage tips\n\n- One can use [`GroundingDinoProcessor`] to prepare image-text pairs for the model.\n- To separate classes in the text use a period e.g. \"a cat. a dog.\"\n- When using multiple classes (e.g. `\"a cat. a dog.\"`), use `post_process_grounded_object_detection` from [`GroundingDinoProcessor`] to post process outputs. Since, the labels returned from `post_process_object_detection` represent the indices from the model dimension where prob > threshold.\n\nHere's how to use the model for zero-shot object detection:\n\n```python\n>>> import requests\n\n>>> import torch\n>>> from PIL import Image\n>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection\n\n>>> model_id = \"IDEA-Research/grounding-dino-tiny\"\n>>> device = \"cuda\"\n\n>>> processor = AutoProcessor.from_pretrained(model_id)\n>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id).to(device)\n\n>>> image_url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(image_url, stream=True).raw)\n>>> # Check for cats and remote controls\n>>> text = \"a cat. a remote control.\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 46c770f1-7e97-4bac-90e0-e26f71ef7884
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"46c770f1-7e97-4bac-90e0-e26f71ef7884\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c492a720-2872-4241-aa10-3bfc474c2143\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2eb4d165131951bb06541e07ce3f2b388d9f65af14dc2bae06ed20b902632072\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7cbd9cc1-749b-4351-b2d1-8522d58f3334\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"067e65b7455248c324e507e6f8c63922352c2dbe43ab2716650caf265854714f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"63e87193-e232-4831-812e-33620db9b322\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"38840528246ff7cd0f6775955bb289d6e21230694760df90abdad62daa72bd63\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1959, \"end_char_idx\": 4651, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c492a720-2872-4241-aa10-3bfc474c2143", "doc_id": "c492a720-2872-4241-aa10-3bfc474c2143", "ref_doc_id": "c492a720-2872-4241-aa10-3bfc474c2143", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2303.05499\">original paper</a>. </small>\n\nThis model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/IDEA-Research/GroundingDINO).\n\n## Usage tips\n\n- One can use [`GroundingDinoProcessor`] to prepare image-text pairs for the model.\n- To separate classes in the text use a period e.g. \"a cat. a dog.\"\n- When using multiple classes (e.g. `\"a cat. a dog.\"`), use `post_process_grounded_object_detection` from [`GroundingDinoProcessor`] to post process outputs. Since, the labels returned from `post_process_object_detection` represent the indices from the model dimension where prob > threshold.\n\nHere's how to use the model for zero-shot object detection:\n\n```python\n>>> import requests\n\n>>> import torch\n>>> from PIL import Image\n>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection\n\n>>> model_id = \"IDEA-Research/grounding-dino-tiny\"\n>>> device = \"cuda\"\n\n>>> processor = AutoProcessor.from_pretrained(model_id)\n>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id).to(device)\n\n>>> image_url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(image_url, stream=True).raw)\n>>> # Check for cats and remote controls\n>>> text = \"a cat. a remote control.\"\n\n>>> inputs = processor(images=image, text=text, return_tensors=\"pt\").to(device)\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> results = processor.post_process_grounded_object_detection(\n...     outputs,\n...     inputs.input_ids,\n...     box_threshold=0.4,\n...     text_threshold=0.3,\n...     target_sizes=[image.size[::-1]]\n... )\n>>> print(results)\n[{'boxes': tensor([[344.6959,  23.1090, 637.1833, 374.2751],\n        [ 12.2666,  51.9145, 316.8582, 472.4392],\n        [ 38.5742,  70.0015, 176.7838, 118.1806]], device='cuda:0'),\n  'labels': ['a cat', 'a cat', 'a remote control'],\n  'scores': tensor([0.4785, 0.4381, 0.4776], device='cuda:0')}]\n```\n\n## Grounded SAM\n\nOne can combine Grounding DINO with the [Segment Anything](sam) model for text-based mask generation as introduced in [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159). You can refer to this [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Grounding%20DINO/GroundingDINO_with_Segment_Anything.ipynb) \ud83c\udf0d for details.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grounded_sam.png\"\nalt=\"drawing\" width=\"900\"/>\n\n<small> Grounded SAM overview."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 63e87193-e232-4831-812e-33620db9b322
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"63e87193-e232-4831-812e-33620db9b322\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c492a720-2872-4241-aa10-3bfc474c2143\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2eb4d165131951bb06541e07ce3f2b388d9f65af14dc2bae06ed20b902632072\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"46c770f1-7e97-4bac-90e0-e26f71ef7884\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1abb708de863b15b7b00d1f6928013c01c56cd4fa1e6316e4f8901c22c6f9410\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3355, \"end_char_idx\": 5788, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c492a720-2872-4241-aa10-3bfc474c2143", "doc_id": "c492a720-2872-4241-aa10-3bfc474c2143", "ref_doc_id": "c492a720-2872-4241-aa10-3bfc474c2143", "chroma:document": "a remote control.\"\n\n>>> inputs = processor(images=image, text=text, return_tensors=\"pt\").to(device)\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> results = processor.post_process_grounded_object_detection(\n...     outputs,\n...     inputs.input_ids,\n...     box_threshold=0.4,\n...     text_threshold=0.3,\n...     target_sizes=[image.size[::-1]]\n... )\n>>> print(results)\n[{'boxes': tensor([[344.6959,  23.1090, 637.1833, 374.2751],\n        [ 12.2666,  51.9145, 316.8582, 472.4392],\n        [ 38.5742,  70.0015, 176.7838, 118.1806]], device='cuda:0'),\n  'labels': ['a cat', 'a cat', 'a remote control'],\n  'scores': tensor([0.4785, 0.4381, 0.4776], device='cuda:0')}]\n```\n\n## Grounded SAM\n\nOne can combine Grounding DINO with the [Segment Anything](sam) model for text-based mask generation as introduced in [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159). You can refer to this [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Grounding%20DINO/GroundingDINO_with_Segment_Anything.ipynb) \ud83c\udf0d for details.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grounded_sam.png\"\nalt=\"drawing\" width=\"900\"/>\n\n<small> Grounded SAM overview. Taken from the <a href=\"https://github.com/IDEA-Research/Grounded-Segment-Anything\">original repository</a>. </small>\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Grounding DINO. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Demo notebooks regarding inference with Grounding DINO as well as combining it with [SAM](sam) can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Grounding%20DINO). \ud83c\udf0e\n\n## GroundingDinoImageProcessor\n\n[[autodoc]] GroundingDinoImageProcessor\n    - preprocess\n    - post_process_object_detection\n\n## GroundingDinoProcessor\n\n[[autodoc]] GroundingDinoProcessor\n    - post_process_grounded_object_detection\n\n## GroundingDinoConfig\n\n[[autodoc]] GroundingDinoConfig\n\n## GroundingDinoModel\n\n[[autodoc]] GroundingDinoModel\n    - forward\n\n## GroundingDinoForObjectDetection\n\n[[autodoc]] GroundingDinoForObjectDetection\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 64364447-6fa3-4324-86b3-2f658e6acebc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/marian", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"64364447-6fa3-4324-86b3-2f658e6acebc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d6fc7042-8e22-4f08-8972-6060b5c5f5c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"59b395f152c4897a2fc06946765f6a4a770f28f72074671bacbb6c29bdecae5f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3598eba7-b0d8-4fe5-b0e3-50aed8a06ee9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"00549dccda6ac18f7182823f17a5f89a9216ec2cf4ea2261346c11eadd9215f7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3075, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "doc_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "ref_doc_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "chroma:document": "# MarianMT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=marian\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-marian-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/opus-mt-zh-en\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nA framework for translation models, using the same models as BART. Translations should be similar, but not identical to output in the test set linked to in each model card.\nThis model was contributed by [sshleifer](https://huggingface.co/sshleifer).\n\n\n## Implementation Notes\n\n- Each model is about 298 MB on disk, there are more than 1,000 models.\n- The list of supported language pairs can be found [here](https://huggingface.co/Helsinki-NLP).\n- Models were originally trained by [J\u00f6rg Tiedemann](https://researchportal.helsinki.fi/en/persons/j%C3%B6rg-tiedemann) using the [Marian](https://marian-nmt.github.io/) C++ library, which supports fast training and translation.\n- All models are transformer encoder-decoders with 6 layers in each component. Each model's performance is documented\n  in a model card.\n- The 80 opus models that require BPE preprocessing are not supported.\n- The modeling code is the same as [`BartForConditionalGeneration`] with a few minor modifications:\n\n  - static (sinusoid) positional embeddings (`MarianConfig.static_position_embeddings=True`)\n  - no layernorm_embedding (`MarianConfig.normalize_embedding=False`)\n  - the model starts generating with `pad_token_id` (which has 0 as a token_embedding) as the prefix (Bart uses\n    `<s/>`),\n- Code to bulk convert models can be found in `convert_marian_to_pytorch.py`.\n\n\n## Naming\n\n- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`\n- The language codes used to name models are inconsistent. Two digit codes can usually be found [here](https://developers.google.com/admin-sdk/directory/v1/languages), three digit codes require googling \"language\n  code {code}\".\n- Codes formatted like `es_AR` are usually `code_{region}`. That one is Spanish from Argentina.\n- The models were converted in two stages. The first 1000 models use ISO-639-2 codes to identify languages, the second\n  group use a combination of ISO-639-5 codes and ISO-639-2 codes.\n\n\n## Examples\n\n- Since Marian models are smaller than many other translation models available in the library, they can be useful for\n  fine-tuning experiments and integration tests.\n- [Fine-tune on GPU](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/train_distil_marian_enro.sh)\n\n## Multilingual Models\n\n- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`:\n- If a model can output multiple languages, and you should specify a language code by prepending the desired output\n  language to the `src_text`.\n- You can see a models's supported language codes in its model card, under target constituents, like in [opus-mt-en-roa](https://huggingface.co/Helsinki-NLP/opus-mt-en-roa)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3598eba7-b0d8-4fe5-b0e3-50aed8a06ee9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/marian", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3598eba7-b0d8-4fe5-b0e3-50aed8a06ee9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d6fc7042-8e22-4f08-8972-6060b5c5f5c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"59b395f152c4897a2fc06946765f6a4a770f28f72074671bacbb6c29bdecae5f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"64364447-6fa3-4324-86b3-2f658e6acebc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8202e6b3c5af673240c275c927cfd0901fdb3f6e6ae9ded998269aa6bd3df14e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"717dd2c9-8ec7-4525-9684-6ac1ae98ec0f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b763b721895cd79422aa777b710922e81e61a50d45b95a0986078267d9497f71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1739, \"end_char_idx\": 4290, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "doc_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "ref_doc_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "chroma:document": "## Naming\n\n- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`\n- The language codes used to name models are inconsistent. Two digit codes can usually be found [here](https://developers.google.com/admin-sdk/directory/v1/languages), three digit codes require googling \"language\n  code {code}\".\n- Codes formatted like `es_AR` are usually `code_{region}`. That one is Spanish from Argentina.\n- The models were converted in two stages. The first 1000 models use ISO-639-2 codes to identify languages, the second\n  group use a combination of ISO-639-5 codes and ISO-639-2 codes.\n\n\n## Examples\n\n- Since Marian models are smaller than many other translation models available in the library, they can be useful for\n  fine-tuning experiments and integration tests.\n- [Fine-tune on GPU](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/train_distil_marian_enro.sh)\n\n## Multilingual Models\n\n- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`:\n- If a model can output multiple languages, and you should specify a language code by prepending the desired output\n  language to the `src_text`.\n- You can see a models's supported language codes in its model card, under target constituents, like in [opus-mt-en-roa](https://huggingface.co/Helsinki-NLP/opus-mt-en-roa).\n- Note that if a model is only multilingual on the source side, like `Helsinki-NLP/opus-mt-roa-en`, no language\n  codes are required.\n\nNew multi-lingual models from the [Tatoeba-Challenge repo](https://github.com/Helsinki-NLP/Tatoeba-Challenge)\nrequire 3 character language codes:\n\n```python\n>>> from transformers import MarianMTModel, MarianTokenizer\n\n>>> src_text = [\n...     \">>fra<< this is a sentence in english that we want to translate to french\",\n...     \">>por<< This should go to portuguese\",\n...     \">>esp<< And this to Spanish\",\n... ]\n\n>>> model_name = \"Helsinki-NLP/opus-mt-en-roa\"\n>>> tokenizer = MarianTokenizer.from_pretrained(model_name)\n>>> print(tokenizer.supported_language_codes)\n['>>zlm_Latn<<', '>>mfe<<', '>>hat<<', '>>pap<<', '>>ast<<', '>>cat<<', '>>ind<<', '>>glg<<', '>>wln<<', '>>spa<<', '>>fra<<', '>>ron<<', '>>por<<', '>>ita<<', '>>oci<<', '>>arg<<', '>>min<<']\n\n>>> model = MarianMTModel.from_pretrained(model_name)\n>>> translated = model.generate(**tokenizer(src_text, return_tensors=\"pt\", padding=True))\n>>> [tokenizer.decode(t, skip_special_tokens=True) for t in translated]\n[\"c'est une phrase en anglais que nous voulons traduire en fran\u00e7ais\",\n 'Isto deve ir para o portugu\u00eas."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 717dd2c9-8ec7-4525-9684-6ac1ae98ec0f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/marian", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"717dd2c9-8ec7-4525-9684-6ac1ae98ec0f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d6fc7042-8e22-4f08-8972-6060b5c5f5c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"59b395f152c4897a2fc06946765f6a4a770f28f72074671bacbb6c29bdecae5f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3598eba7-b0d8-4fe5-b0e3-50aed8a06ee9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"46cad0a4491d1c19323b2f8ab67d45feb8cf6a244936f83555b5bb6c66b1f2a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"28010af2-393c-416b-83ad-b8d2cb10e068\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9e8e09492690d1c849851363e67b89752b7278f8fc4d307fe4fff7848cbf4a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2902, \"end_char_idx\": 6221, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "doc_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "ref_doc_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "chroma:document": "- You can see a models's supported language codes in its model card, under target constituents, like in [opus-mt-en-roa](https://huggingface.co/Helsinki-NLP/opus-mt-en-roa).\n- Note that if a model is only multilingual on the source side, like `Helsinki-NLP/opus-mt-roa-en`, no language\n  codes are required.\n\nNew multi-lingual models from the [Tatoeba-Challenge repo](https://github.com/Helsinki-NLP/Tatoeba-Challenge)\nrequire 3 character language codes:\n\n```python\n>>> from transformers import MarianMTModel, MarianTokenizer\n\n>>> src_text = [\n...     \">>fra<< this is a sentence in english that we want to translate to french\",\n...     \">>por<< This should go to portuguese\",\n...     \">>esp<< And this to Spanish\",\n... ]\n\n>>> model_name = \"Helsinki-NLP/opus-mt-en-roa\"\n>>> tokenizer = MarianTokenizer.from_pretrained(model_name)\n>>> print(tokenizer.supported_language_codes)\n['>>zlm_Latn<<', '>>mfe<<', '>>hat<<', '>>pap<<', '>>ast<<', '>>cat<<', '>>ind<<', '>>glg<<', '>>wln<<', '>>spa<<', '>>fra<<', '>>ron<<', '>>por<<', '>>ita<<', '>>oci<<', '>>arg<<', '>>min<<']\n\n>>> model = MarianMTModel.from_pretrained(model_name)\n>>> translated = model.generate(**tokenizer(src_text, return_tensors=\"pt\", padding=True))\n>>> [tokenizer.decode(t, skip_special_tokens=True) for t in translated]\n[\"c'est une phrase en anglais que nous voulons traduire en fran\u00e7ais\",\n 'Isto deve ir para o portugu\u00eas.',\n 'Y esto al espa\u00f1ol']\n```\n\nHere is the code to see all available pretrained models on the hub:\n\n```python\nfrom huggingface_hub import list_models\n\nmodel_list = list_models()\norg = \"Helsinki-NLP\"\nmodel_ids = [x.id for x in model_list if x.id.startswith(org)]\nsuffix = [x.split(\"/\")[1] for x in model_ids]\nold_style_multi_models = [f\"{org}/{s}\" for s in suffix if s != s.lower()]\n```\n\n## Old Style Multi-Lingual Models\n\nThese are the old style multi-lingual models ported from the OPUS-MT-Train repo: and the members of each language\ngroup:\n\n```python no-style\n['Helsinki-NLP/opus-mt-NORTH_EU-NORTH_EU',\n 'Helsinki-NLP/opus-mt-ROMANCE-en',\n 'Helsinki-NLP/opus-mt-SCANDINAVIA-SCANDINAVIA',\n 'Helsinki-NLP/opus-mt-de-ZH',\n 'Helsinki-NLP/opus-mt-en-CELTIC',\n 'Helsinki-NLP/opus-mt-en-ROMANCE',\n 'Helsinki-NLP/opus-mt-es-NORWAY',\n 'Helsinki-NLP/opus-mt-fi-NORWAY',\n 'Helsinki-NLP/opus-mt-fi-ZH',\n 'Helsinki-NLP/opus-mt-fi_nb_no_nn_ru_sv_en-SAMI',\n 'Helsinki-NLP/opus-mt-sv-NORWAY',\n 'Helsinki-NLP/opus-mt-sv-ZH']\nGROUP_MEMBERS = {\n 'ZH': ['cmn', 'cn', 'yue', 'ze_zh', 'zh_cn', 'zh_CN', 'zh_HK', 'zh_tw', 'zh_TW', 'zh_yue', 'zhs', 'zht', 'zh'],\n 'ROMANCE': ['fr', 'fr_BE', 'fr_CA', 'fr_FR', 'wa', 'frp', 'oc', 'ca', 'rm', 'lld', 'fur', 'lij', 'lmo', 'es', 'es_AR', 'es_CL', 'es_CO', 'es_CR', 'es_DO', 'es_EC', 'es_ES', 'es_GT', 'es_HN', 'es_MX', 'es_NI', 'es_PA', 'es_PE', 'es_PR', 'es_SV', 'es_UY', 'es_VE', 'pt', 'pt_br', 'pt_BR', 'pt_PT', 'gl', 'lad', 'an', 'mwl', 'it', 'it_IT', 'co', 'nap', 'scn', 'vec', 'sc', 'ro', 'la'],\n 'NORTH_EU': ['de', 'nl', 'fy', 'af', 'da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],\n 'SCANDINAVIA': ['da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],\n 'SAMI': ['se', 'sma', 'smj', 'smn', 'sms'],\n 'NORWAY': ['nb_NO', 'nb', 'nn_NO', 'nn', 'nog', 'no_nb', 'no'],\n 'CELTIC': ['ga', 'cy', 'br', 'gd', 'kw', 'gv']\n}\n```\n\nExample of translating english to many romance languages, using old-style 2 character language codes"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 28010af2-393c-416b-83ad-b8d2cb10e068
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/marian", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"28010af2-393c-416b-83ad-b8d2cb10e068\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d6fc7042-8e22-4f08-8972-6060b5c5f5c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"59b395f152c4897a2fc06946765f6a4a770f28f72074671bacbb6c29bdecae5f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"717dd2c9-8ec7-4525-9684-6ac1ae98ec0f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d5304e6f618d7fe32e47daea43af865a8cae3ecbb2bf3bcfb102c9173e67b61e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6224, \"end_char_idx\": 7781, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "doc_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "ref_doc_id": "d6fc7042-8e22-4f08-8972-6060b5c5f5c0", "chroma:document": "```python\n>>> from transformers import MarianMTModel, MarianTokenizer\n\n>>> src_text = [\n...     \">>fr<< this is a sentence in english that we want to translate to french\",\n...     \">>pt<< This should go to portuguese\",\n...     \">>es<< And this to Spanish\",\n... ]\n\n>>> model_name = \"Helsinki-NLP/opus-mt-en-ROMANCE\"\n>>> tokenizer = MarianTokenizer.from_pretrained(model_name)\n\n>>> model = MarianMTModel.from_pretrained(model_name)\n>>> translated = model.generate(**tokenizer(src_text, return_tensors=\"pt\", padding=True))\n>>> tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]\n[\"c'est une phrase en anglais que nous voulons traduire en fran\u00e7ais\", \n 'Isto deve ir para o portugu\u00eas.',\n 'Y esto al espa\u00f1ol']\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## MarianConfig\n\n[[autodoc]] MarianConfig\n\n## MarianTokenizer\n\n[[autodoc]] MarianTokenizer\n    - build_inputs_with_special_tokens\n\n<frameworkcontent>\n<pt>\n\n## MarianModel\n\n[[autodoc]] MarianModel\n    - forward\n\n## MarianMTModel\n\n[[autodoc]] MarianMTModel\n    - forward\n\n## MarianForCausalLM\n\n[[autodoc]] MarianForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFMarianModel\n\n[[autodoc]] TFMarianModel\n    - call\n\n## TFMarianMTModel\n\n[[autodoc]] TFMarianMTModel\n    - call\n\n</tf>\n<jax>\n\n## FlaxMarianModel\n\n[[autodoc]] FlaxMarianModel\n    - __call__\n\n## FlaxMarianMTModel\n\n[[autodoc]] FlaxMarianMTModel\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3c0f5fff-3258-4b87-8db4-5e7d71d4b8c7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vision-text-dual-encoder", "title": "VisionTextDualEncoder", "tokens": 543, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3c0f5fff-3258-4b87-8db4-5e7d71d4b8c7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-text-dual-encoder\", \"title\": \"VisionTextDualEncoder\", \"tokens\": 543, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6e51e1a8-3fd1-4956-93fe-137f715679e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-text-dual-encoder\", \"title\": \"VisionTextDualEncoder\", \"tokens\": 543, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3000cf1f4755f90a5e35c8239ae82fd1827747e86ab9e520bb68b9ed6f955df2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1579, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6e51e1a8-3fd1-4956-93fe-137f715679e1", "doc_id": "6e51e1a8-3fd1-4956-93fe-137f715679e1", "ref_doc_id": "6e51e1a8-3fd1-4956-93fe-137f715679e1", "chroma:document": "# VisionTextDualEncoder\n\n## Overview\n\nThe [`VisionTextDualEncoderModel`] can be used to initialize a vision-text dual encoder model with\nany pretrained vision autoencoding model as the vision encoder (*e.g.* [ViT](vit), [BEiT](beit), [DeiT](deit)) and any pretrained text autoencoding model as the text encoder (*e.g.* [RoBERTa](roberta), [BERT](bert)). Two projection layers are added on top of both the vision and text encoder to project the output embeddings\nto a shared latent space. The projection layers are randomly initialized so the model should be fine-tuned on a\ndownstream task. This model can be used to align the vision-text embeddings using CLIP like contrastive image-text\ntraining and then can be used for zero-shot vision tasks such image-classification or retrieval.\n\nIn [LiT: Zero-Shot Transfer with Locked-image Text Tuning](https://arxiv.org/abs/2111.07991) it is shown how\nleveraging pre-trained (locked/frozen) image and text model for contrastive learning yields significant improvement on\nnew zero-shot vision tasks such as image classification or retrieval.\n\n## VisionTextDualEncoderConfig\n\n[[autodoc]] VisionTextDualEncoderConfig\n\n## VisionTextDualEncoderProcessor\n\n[[autodoc]] VisionTextDualEncoderProcessor\n\n<frameworkcontent>\n<pt>\n\n## VisionTextDualEncoderModel\n\n[[autodoc]] VisionTextDualEncoderModel\n    - forward\n\n</pt>\n<tf>\n\n## FlaxVisionTextDualEncoderModel\n\n[[autodoc]] FlaxVisionTextDualEncoderModel\n    - __call__\n\n</tf>\n<jax>\n\n## TFVisionTextDualEncoderModel\n\n[[autodoc]] TFVisionTextDualEncoderModel\n    - call\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 979f7a67-c087-42a5-a0fa-67adc6b4c9ed
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"979f7a67-c087-42a5-a0fa-67adc6b4c9ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e44faccf-cec5-4da6-a4ac-b4e0a633efee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4b2f68ccd663b927abd29ec9b531301ca0f06c1b5b18a2d1502343ff08cb7762\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4244b4d2-21f2-4754-a02c-441faaab7027\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2d00388786c48a4f235b7d2e610f12909bbbfd51ff605d8952ed50a732463673\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3248, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "doc_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "ref_doc_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "chroma:document": "# NLLB-MOE\n\n\n## Overview\n\nThe NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-juss\u00e0, James Cross, Onur \u00c7elebi,\nMaha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,\nLoic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,\nNecip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm\u00e1n, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,\nSafiyyah Saleem, Holger Schwenk, and Jeff Wang.\n\nThe abstract of the paper is the following:\n\n*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.\nHowever, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the\n200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind? In No Language Left Behind, we took on this challenge by\nfirst contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed\nat narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of\nExperts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training\nimprovements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using\na human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.\nOur model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/facebookresearch/fairseq).\n\n## Usage tips\n\n- M2M100ForConditionalGeneration is the base model for both NLLB and NLLB MoE\n- The NLLB-MoE is very similar to the NLLB model, but it's feed forward layer is based on the implementation of SwitchTransformers.\n- The tokenizer is the same as the NLLB models.\n\n## Implementation differences with SwitchTransformers\n\nThe biggest difference is the way the tokens are routed. NLLB-MoE uses a `top-2-gate` which means that for each input, only the top two experts are selected based on the \nhighest predicted probabilities from the gating network, and the remaining experts are ignored. In `SwitchTransformers`, only the top-1 probabilities are computed, \nwhich means that tokens have less probability of being forwarded."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4244b4d2-21f2-4754-a02c-441faaab7027
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4244b4d2-21f2-4754-a02c-441faaab7027\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e44faccf-cec5-4da6-a4ac-b4e0a633efee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4b2f68ccd663b927abd29ec9b531301ca0f06c1b5b18a2d1502343ff08cb7762\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"979f7a67-c087-42a5-a0fa-67adc6b4c9ed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"26f4672926378398a9e76b997f42b7a6210a7c971a617c737837756b1ff62ea7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b2f8198-0513-4428-8bdb-21519bcfcb47\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"385c780de5a64533d87c0100236afa3bd959ff47ba979011d2e4c1060ad2c756\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1457, \"end_char_idx\": 4426, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "doc_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "ref_doc_id": "e44faccf-cec5-4da6-a4ac-b4e0a633efee", "chroma:document": "Then, we created datasets and models aimed\nat narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of\nExperts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training\nimprovements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using\na human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.\nOur model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/facebookresearch/fairseq).\n\n## Usage tips\n\n- M2M100ForConditionalGeneration is the base model for both NLLB and NLLB MoE\n- The NLLB-MoE is very similar to the NLLB model, but it's feed forward layer is based on the implementation of SwitchTransformers.\n- The tokenizer is the same as the NLLB models.\n\n## Implementation differences with SwitchTransformers\n\nThe biggest difference is the way the tokens are routed. NLLB-MoE uses a `top-2-gate` which means that for each input, only the top two experts are selected based on the \nhighest predicted probabilities from the gating network, and the remaining experts are ignored. In `SwitchTransformers`, only the top-1 probabilities are computed, \nwhich means that tokens have less probability of being forwarded. Moreover, if a token is not routed to any expert, `SwitchTransformers` still adds its unmodified hidden \nstates (kind of like a residual connection) while they are masked in `NLLB`'s top-2 routing mechanism. \n\n## Generating with NLLB-MoE\n\nThe available checkpoints require around 350GB of storage. Make sure to use `accelerate` if you do not have enough RAM on your machine.\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-moe-54b\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-moe-54b\")\n\n>>> article = \"Previously, Ring's CEO, Jamie Siminoff, remarked the company started when his doorbell wasn't audible from his shop in his garage.\""}


