# embeddings_queue

created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 44a5f27e-f297-4f07-92ed-0be0f4127372
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"44a5f27e-f297-4f07-92ed-0be0f4127372\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4522d739-c55b-49ba-953c-f3626a4aec16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0f43936d-6186-431a-ac96-f744574e2e7d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"94e263657d40bad369389e7e2eae36785ad87fcc6599bb5c6ab5eb6ae6e2be8a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b2ad812-dbcc-432f-a467-b5910038e677\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8c9b609ffaa33c00e18c148884ad24c43adb5f9aa490f301120f904ce5aceecf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3525, \"end_char_idx\": 6715, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "ref_doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "chroma:document": "```python\n>>> from PIL import Image\n>>> import requests\n\n>>> from transformers import CLIPProcessor, CLIPModel\n\n>>> model = CLIPModel.from_pretrained(\"openai/clip-vit-base-patch32\")\n>>> processor = CLIPProcessor.from_pretrained(\"openai/clip-vit-base-patch32\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> inputs = processor(text=[\"a photo of a cat\", \"a photo of a dog\"], images=image, return_tensors=\"pt\", padding=True)\n\n>>> outputs = model(**inputs)\n>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score\n>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities\n```\n\n\n### Combining CLIP and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\n<Tip warning={true}>\n\nFor small batch sizes, you might notice a slowdown in your model when using flash attention. Refer to the section [Expected speedups with Flash Attention and SDPA](#Expected-speedups-with-Flash-Attention-and-SDPA) below and select an appropriate attention implementation.\n\n</Tip>\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> import requests\n>>> from PIL import Image\n\n>>> from transformers import CLIPProcessor, CLIPModel\n\n>>> device = \"cuda\"\n>>> torch_dtype = torch.float16\n\n>>> model = CLIPModel.from_pretrained(\n...     \"openai/clip-vit-base-patch32\",\n...     attn_implementation=\"flash_attention_2\",\n...     device_map=device,\n...     torch_dtype=torch_dtype,\n... )\n>>> processor = CLIPProcessor.from_pretrained(\"openai/clip-vit-base-patch32\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> inputs = processor(text=[\"a photo of a cat\", \"a photo of a dog\"], images=image, return_tensors=\"pt\", padding=True)\n>>> inputs.to(device)\n\n>>> with torch.no_grad():\n...     with torch.autocast(device):\n...         outputs = model(**inputs)\n\n>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score\n>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities\n>>> print(probs)\ntensor([[0.9946, 0.0052]], device='cuda:0', dtype=torch.float16)\n```\n\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4b2ad812-dbcc-432f-a467-b5910038e677
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4b2ad812-dbcc-432f-a467-b5910038e677\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4522d739-c55b-49ba-953c-f3626a4aec16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"44a5f27e-f297-4f07-92ed-0be0f4127372\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d053d7668e2b8edfe7357b56dc60742838e5d4762e6f5b4aaf30eb70e90d4785\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5b6ddb84-b686-4d96-a6b8-d35fefc05ce7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52da01a60c34814bc5363d1e8e3352c2f893e8e81bcac1d0ac62ec43a933a6f7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6156, \"end_char_idx\": 8282, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "ref_doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "chroma:document": "### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import CLIPModel\n\nmodel = CLIPModel.from_pretrained(\"openai/clip-vit-base-patch32\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\n### Expected speedups with Flash Attention and SDPA\n\nOn a local benchmark (NVIDIA A10G, PyTorch 2.3.1+cu121) with `float16`, we saw the following speedups during inference for `\"openai/clip-vit-large-patch14\"` checkpoint ([code](https://gist.github.com/qubvel/ac691a54e54f9fae8144275f866a7ff8)):\n\n#### CLIPTextModel\n\n|   Num text labels |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                 4 |            0.009 |          0.012 |         0.737 |           0.007 |          1.269 |\n|                16 |            0.009 |          0.014 |         0.659 |           0.008 |          1.187 |\n|                32 |            0.018 |          0.021 |         0.862 |           0.016 |          1.142 |\n|                64 |            0.034 |          0.034 |         1.001 |           0.03  |          1.163 |\n|               128 |            0.063 |          0.058 |         1.09  |           0.054 |          1.174 |\n\n!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5b6ddb84-b686-4d96-a6b8-d35fefc05ce7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5b6ddb84-b686-4d96-a6b8-d35fefc05ce7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4522d739-c55b-49ba-953c-f3626a4aec16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4b2ad812-dbcc-432f-a467-b5910038e677\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"81397ee65a5facb4c0e79eda0373bd9e270fbc64523048e141e81b47c6904dfc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"56ca8b0d-7da3-4d4b-afd5-9fdc183ce855\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3e258336f1bf3c9df50ab201ba1d8fa3c945905a818c294594b7f92f29f4d26b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7161, \"end_char_idx\": 9072, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "ref_doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "chroma:document": "`torch.float16` or `torch.bfloat16`).\n\n### Expected speedups with Flash Attention and SDPA\n\nOn a local benchmark (NVIDIA A10G, PyTorch 2.3.1+cu121) with `float16`, we saw the following speedups during inference for `\"openai/clip-vit-large-patch14\"` checkpoint ([code](https://gist.github.com/qubvel/ac691a54e54f9fae8144275f866a7ff8)):\n\n#### CLIPTextModel\n\n|   Num text labels |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                 4 |            0.009 |          0.012 |         0.737 |           0.007 |          1.269 |\n|                16 |            0.009 |          0.014 |         0.659 |           0.008 |          1.187 |\n|                32 |            0.018 |          0.021 |         0.862 |           0.016 |          1.142 |\n|                64 |            0.034 |          0.034 |         1.001 |           0.03  |          1.163 |\n|               128 |            0.063 |          0.058 |         1.09  |           0.054 |          1.174 |\n\n![clip_text_model_viz_3](https://github.com/user-attachments/assets/e9826b43-4e66-4f4c-952b-af4d90bd38eb)\n\n#### CLIPVisionModel\n\n|   Image batch size |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|-------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                  1 |            0.016 |          0.013 |         1.247 |           0.012 |          1.318 |\n|                  4 |            0.025 |          0.021 |         1.198 |           0.021 |          1.202 |\n|                 16 |            0.093 |          0.075 |         1.234 |           0.075 |          1.24  |\n|                 32 |            0.181 |          0.147 |         1.237 |           0.146 |          1.241 |\n\n!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 56ca8b0d-7da3-4d4b-afd5-9fdc183ce855
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"56ca8b0d-7da3-4d4b-afd5-9fdc183ce855\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4522d739-c55b-49ba-953c-f3626a4aec16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5b6ddb84-b686-4d96-a6b8-d35fefc05ce7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a3616684fe5bb8f096bda617d6142ec1a406bf5dc9001070e52255a3fba4e7a9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"69801ed4-475c-41f2-976e-8271792b08b5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eb72e2bfcb67fdec0622a02fda7064d4fcff0228fc7028797017f946b95a1a04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8282, \"end_char_idx\": 11137, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "ref_doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "chroma:document": "[clip_text_model_viz_3](https://github.com/user-attachments/assets/e9826b43-4e66-4f4c-952b-af4d90bd38eb)\n\n#### CLIPVisionModel\n\n|   Image batch size |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|-------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                  1 |            0.016 |          0.013 |         1.247 |           0.012 |          1.318 |\n|                  4 |            0.025 |          0.021 |         1.198 |           0.021 |          1.202 |\n|                 16 |            0.093 |          0.075 |         1.234 |           0.075 |          1.24  |\n|                 32 |            0.181 |          0.147 |         1.237 |           0.146 |          1.241 |\n\n![clip_image_model_viz_3](https://github.com/user-attachments/assets/50a36206-e3b9-4adc-ac8e-926b8b071d63)\n\n#### CLIPModel\n\n|   Image batch size |   Num text labels |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|-------------------:|------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                  1 |                 4 |            0.025 |          0.026 |         0.954 |           0.02  |          1.217 |\n|                  1 |                16 |            0.026 |          0.028 |         0.918 |           0.02  |          1.287 |\n|                  1 |                64 |            0.042 |          0.046 |         0.906 |           0.036 |          1.167 |\n|                  4 |                 4 |            0.028 |          0.033 |         0.849 |           0.024 |          1.189 |\n|                  4 |                16 |            0.034 |          0.035 |         0.955 |           0.029 |          1.169 |\n|                  4 |                64 |            0.059 |          0.055 |         1.072 |           0.05  |          1.179 |\n|                 16 |                 4 |            0.096 |          0.088 |         1.091 |           0.078 |          1.234 |\n|                 16 |                16 |            0.102 |          0.09  |         1.129 |           0.083 |          1.224 |\n|                 16 |                64 |            0.127 |          0.11  |         1.157 |           0.105 |          1.218 |\n|                 32 |                 4 |            0.185 |          0.159 |         1.157 |           0.149 |          1.238 |\n|                 32 |                16 |            0.19  |          0.162 |         1.177 |           0.154 |          1.233 |\n|                 32 |                64 |            0.216 |          0.181 |         1.19  |           0.176 |          1.228 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CLIP."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 69801ed4-475c-41f2-976e-8271792b08b5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"69801ed4-475c-41f2-976e-8271792b08b5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4522d739-c55b-49ba-953c-f3626a4aec16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"56ca8b0d-7da3-4d4b-afd5-9fdc183ce855\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"791b7f8bfddfa2ac95576f50d51180ee7425a77a9cc99ee445f1a281a22025d1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2395a683-8b38-4ecb-9501-9c421c9d89e4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"596220c249636460c7095edfb37084f21d7f4482dfe4c66836377ac426b2cec0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11139, \"end_char_idx\": 13263, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "ref_doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "chroma:document": "- [Fine tuning CLIP with Remote Sensing (Satellite) images and captions](https://huggingface.co/blog/fine-tune-clip-rsicd), a blog post about how to fine-tune CLIP with [RSICD dataset](https://github.com/201528014227051/RSICD_optimal) and comparison of performance changes due to data augmentation.\n- This [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text) shows how to train a CLIP-like vision-text dual encoder model using a pre-trained vision and text encoder using [COCO dataset](https://cocodataset.org/#home).\n\n<PipelineTag pipeline=\"image-to-text\"/>\n\n- A [notebook](https://colab.research.google.com/drive/1tuoAC5F4sC7qid56Z0ap-stR3rwdk0ZV?usp=sharing) on how to use a pretrained CLIP for inference with beam search for image captioning. \ud83c\udf0e\n\n**Image retrieval**\n\n- A [notebook](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing) on image retrieval using pretrained CLIP and computing MRR(Mean Reciprocal Rank) score. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/github/deep-diver/image_search_with_natural_language/blob/main/notebooks/Image_Search_CLIP.ipynb) on image retrieval and showing the similarity score. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/1xO-wC_m_GNzgjIBQ4a4znvQkvDoZJvH4?usp=sharing) on how to map images and texts to the same vector space using Multilingual CLIP. \ud83c\udf0e \n- A [notebook](https://colab.research.google.com/github/vivien000/clip-demo/blob/master/clip.ipynb#scrollTo=uzdFhRGqiWkR) on how to run CLIP on semantic image search using [Unsplash](https://unsplash.com) and [TMDB](https://www.themoviedb.org/) datasets. \ud83c\udf0e\n\n**Explainability**\n\n- A [notebook](https://colab.research.google.com/github/hila-chefer/Transformer-MM-Explainability/blob/main/CLIP_explainability.ipynb) on how to visualize similarity between input token and image segment. \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2395a683-8b38-4ecb-9501-9c421c9d89e4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2395a683-8b38-4ecb-9501-9c421c9d89e4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4522d739-c55b-49ba-953c-f3626a4aec16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"69801ed4-475c-41f2-976e-8271792b08b5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d26bbed2db0a418fddd584e0fcdbbf5c27ba88d4f831f6c0ac417b42adc8c9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11945, \"end_char_idx\": 14972, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "ref_doc_id": "4522d739-c55b-49ba-953c-f3626a4aec16", "chroma:document": "\ud83c\udf0e\n\n**Image retrieval**\n\n- A [notebook](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing) on image retrieval using pretrained CLIP and computing MRR(Mean Reciprocal Rank) score. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/github/deep-diver/image_search_with_natural_language/blob/main/notebooks/Image_Search_CLIP.ipynb) on image retrieval and showing the similarity score. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/1xO-wC_m_GNzgjIBQ4a4znvQkvDoZJvH4?usp=sharing) on how to map images and texts to the same vector space using Multilingual CLIP. \ud83c\udf0e \n- A [notebook](https://colab.research.google.com/github/vivien000/clip-demo/blob/master/clip.ipynb#scrollTo=uzdFhRGqiWkR) on how to run CLIP on semantic image search using [Unsplash](https://unsplash.com) and [TMDB](https://www.themoviedb.org/) datasets. \ud83c\udf0e\n\n**Explainability**\n\n- A [notebook](https://colab.research.google.com/github/hila-chefer/Transformer-MM-Explainability/blob/main/CLIP_explainability.ipynb) on how to visualize similarity between input token and image segment. \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## CLIPConfig\n\n[[autodoc]] CLIPConfig\n    - from_text_vision_configs\n\n## CLIPTextConfig\n\n[[autodoc]] CLIPTextConfig\n\n## CLIPVisionConfig\n\n[[autodoc]] CLIPVisionConfig\n\n## CLIPTokenizer\n\n[[autodoc]] CLIPTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## CLIPTokenizerFast\n\n[[autodoc]] CLIPTokenizerFast\n\n## CLIPImageProcessor\n\n[[autodoc]] CLIPImageProcessor\n    - preprocess\n\n## CLIPFeatureExtractor\n\n[[autodoc]] CLIPFeatureExtractor\n\n## CLIPProcessor\n\n[[autodoc]] CLIPProcessor\n\n<frameworkcontent>\n<pt>\n\n## CLIPModel\n\n[[autodoc]] CLIPModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## CLIPTextModel\n\n[[autodoc]] CLIPTextModel\n    - forward\n\n## CLIPTextModelWithProjection\n\n[[autodoc]] CLIPTextModelWithProjection\n    - forward\n\n## CLIPVisionModelWithProjection\n\n[[autodoc]] CLIPVisionModelWithProjection\n    - forward\n\n## CLIPVisionModel\n\n[[autodoc]] CLIPVisionModel\n    - forward\n\n## CLIPForImageClassification\n\n[[autodoc]] CLIPForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFCLIPModel\n\n[[autodoc]] TFCLIPModel\n    - call\n    - get_text_features\n    - get_image_features\n\n## TFCLIPTextModel\n\n[[autodoc]] TFCLIPTextModel\n    - call\n\n## TFCLIPVisionModel\n\n[[autodoc]] TFCLIPVisionModel\n    - call\n\n</tf>\n<jax>\n\n## FlaxCLIPModel\n\n[[autodoc]] FlaxCLIPModel\n    - __call__\n    - get_text_features\n    - get_image_features\n\n## FlaxCLIPTextModel\n\n[[autodoc]] FlaxCLIPTextModel\n    - __call__\n\n## FlaxCLIPTextModelWithProjection\n\n[[autodoc]] FlaxCLIPTextModelWithProjection\n    - __call__\n\n## FlaxCLIPVisionModel\n\n[[autodoc]] FlaxCLIPVisionModel\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9243b77d-6e1d-4359-8992-35aed689938c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9243b77d-6e1d-4359-8992-35aed689938c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"446bb654-e380-4147-9ab8-e190ec87a045\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4566661e98d9827c0da46dcec5837c948b86667c5cb5005f1b768fe04adc594d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e7754160-4287-4be4-9bbb-d37b2a1e46ea\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"20a1b526ff550aca1c48e43c3a93e72c0b853ce8b92f07de6a3a3e1f2bad3486\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3395, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "446bb654-e380-4147-9ab8-e190ec87a045", "doc_id": "446bb654-e380-4147-9ab8-e190ec87a045", "ref_doc_id": "446bb654-e380-4147-9ab8-e190ec87a045", "chroma:document": "# RT-DETR\n\n## Overview\n\n\nThe RT-DETR model was proposed in [DETRs Beat YOLOs on Real-time Object Detection](https://arxiv.org/abs/2304.08069) by Wenyu Lv, Yian Zhao, Shangliang Xu, Jinman Wei, Guanzhong Wang, Cheng Cui, Yuning Du, Qingqing Dang, Yi Liu.\n\nRT-DETR is an object detection model that stands for \"Real-Time DEtection Transformer.\" This model is designed to perform object detection tasks with a focus on achieving real-time performance while maintaining high accuracy. Leveraging the transformer architecture, which has gained significant popularity in various fields of deep learning, RT-DETR processes images to identify and locate multiple objects within them.\n\nThe abstract from the paper is the following:\n\n*Recently, end-to-end transformer-based detectors (DETRs) have achieved remarkable performance. However, the issue of the high computational cost of DETRs has not been effectively addressed, limiting their practical application and preventing them from fully exploiting the benefits of no post-processing, such as non-maximum suppression (NMS). In this paper, we first analyze the influence of NMS in modern real-time object detectors on inference speed, and establish an end-to-end speed benchmark. To avoid the inference delay caused by NMS, we propose a Real-Time DEtection TRansformer (RT-DETR), the first real-time end-to-end object detector to our best knowledge. Specifically, we design an efficient hybrid encoder to efficiently process multi-scale features by decoupling the intra-scale interaction and cross-scale fusion, and propose IoU-aware query selection to improve the initialization of object queries. In addition, our proposed detector supports flexibly adjustment of the inference speed by using different decoder layers without the need for retraining, which facilitates the practical application of real-time object detectors. Our RT-DETR-L achieves 53.0% AP on COCO val2017 and 114 FPS on T4 GPU, while RT-DETR-X achieves 54.8% AP and 74 FPS, outperforming all YOLO detectors of the same scale in both speed and accuracy. Furthermore, our RT-DETR-R50 achieves 53.1% AP and 108 FPS, outperforming DINO-Deformable-DETR-R50 by 2.2% AP in accuracy and by about 21 times in FPS.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/rt_detr_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> RT-DETR performance relative to YOLO models. Taken from the <a href=\"https://arxiv.org/abs/2304.08069\">original paper.</a> </small>\n\nThe model version was contributed by [rafaelpadilla](https://huggingface.co/rafaelpadilla) and [sangbumchoi](https://github.com/SangbumChoi). The original code can be found [here](https://github.com/lyuwenyu/RT-DETR/).\n\n\n## Usage tips\n\nInitially, an image is processed using a pre-trained convolutional neural network, specifically a Resnet-D variant as referenced in the original code. This network extracts features from the final three layers of the architecture. Following this, a hybrid encoder is employed to convert the multi-scale features into a sequential array of image features. Then, a decoder, equipped with auxiliary prediction heads is used to refine the object queries. This process facilitates the direct generation of bounding boxes, eliminating the need for any additional post-processing to acquire the logits and coordinates for the bounding boxes."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e7754160-4287-4be4-9bbb-d37b2a1e46ea
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e7754160-4287-4be4-9bbb-d37b2a1e46ea\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"446bb654-e380-4147-9ab8-e190ec87a045\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4566661e98d9827c0da46dcec5837c948b86667c5cb5005f1b768fe04adc594d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9243b77d-6e1d-4359-8992-35aed689938c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"96486a55d6008d5a1a1f6d7f5c39cebd379801f071e5c4066738824246b12b9b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1d360c75-a8c7-49ca-b6f6-981191602d7b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"63d76cb9bd536d7c10a59dcb217146198c9293fd5cfc6d8daa9a51a5588c24e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2746, \"end_char_idx\": 5264, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "446bb654-e380-4147-9ab8-e190ec87a045", "doc_id": "446bb654-e380-4147-9ab8-e190ec87a045", "ref_doc_id": "446bb654-e380-4147-9ab8-e190ec87a045", "chroma:document": "## Usage tips\n\nInitially, an image is processed using a pre-trained convolutional neural network, specifically a Resnet-D variant as referenced in the original code. This network extracts features from the final three layers of the architecture. Following this, a hybrid encoder is employed to convert the multi-scale features into a sequential array of image features. Then, a decoder, equipped with auxiliary prediction heads is used to refine the object queries. This process facilitates the direct generation of bounding boxes, eliminating the need for any additional post-processing to acquire the logits and coordinates for the bounding boxes.\n\n```py\n>>> import torch\n>>> import requests\n\n>>> from PIL import Image\n>>> from transformers import RTDetrForObjectDetection, RTDetrImageProcessor\n\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg' \n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> image_processor = RTDetrImageProcessor.from_pretrained(\"PekingU/rtdetr_r50vd\")\n>>> model = RTDetrForObjectDetection.from_pretrained(\"PekingU/rtdetr_r50vd\")\n\n>>> inputs = image_processor(images=image, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> results = image_processor.post_process_object_detection(outputs, target_sizes=torch.tensor([image.size[::-1]]), threshold=0.3)\n\n>>> for result in results:\n...     for score, label_id, box in zip(result[\"scores\"], result[\"labels\"], result[\"boxes\"]):\n...         score, label = score.item(), label_id.item()\n...         box = [round(i, 2) for i in box.tolist()]\n...         print(f\"{model.config.id2label[label]}: {score:.2f} {box}\")\nsofa: 0.97 [0.14, 0.38, 640.13, 476.21]\ncat: 0.96 [343.38, 24.28, 640.14, 371.5]\ncat: 0.96 [13.23, 54.18, 318.98, 472.22]\nremote: 0.95 [40.11, 73.44, 175.96, 118.48]\nremote: 0.92 [333.73, 76.58, 369.97, 186.99]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RT-DETR.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- Scripts for finetuning [`RTDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n- Notebooks regarding inference and fine-tuning RT-DETR on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/RT-DETR)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1d360c75-a8c7-49ca-b6f6-981191602d7b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1d360c75-a8c7-49ca-b6f6-981191602d7b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"446bb654-e380-4147-9ab8-e190ec87a045\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4566661e98d9827c0da46dcec5837c948b86667c5cb5005f1b768fe04adc594d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e7754160-4287-4be4-9bbb-d37b2a1e46ea\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc231773a5887d7cd076c79cc99f779dd2972ca401939b8d0efcdb21b772a96\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4737, \"end_char_idx\": 5685, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "446bb654-e380-4147-9ab8-e190ec87a045", "doc_id": "446bb654-e380-4147-9ab8-e190ec87a045", "ref_doc_id": "446bb654-e380-4147-9ab8-e190ec87a045", "chroma:document": "<PipelineTag pipeline=\"object-detection\"/>\n\n- Scripts for finetuning [`RTDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n- Notebooks regarding inference and fine-tuning RT-DETR on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/RT-DETR). \ud83c\udf0e\n\n## RTDetrConfig\n\n[[autodoc]] RTDetrConfig\n\n## RTDetrResNetConfig\n\n[[autodoc]] RTDetrResNetConfig\n\n## RTDetrImageProcessor\n\n[[autodoc]] RTDetrImageProcessor\n    - preprocess\n    - post_process_object_detection\n\n## RTDetrModel\n\n[[autodoc]] RTDetrModel\n    - forward\n\n## RTDetrForObjectDetection\n\n[[autodoc]] RTDetrForObjectDetection\n    - forward\n\n## RTDetrResNetBackbone\n\n[[autodoc]] RTDetrResNetBackbone\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b592f40a-ca39-477f-92bb-d8f50519daeb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b592f40a-ca39-477f-92bb-d8f50519daeb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ac6893-6ccd-4c84-a098-a776207f8d3e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"edf3f144-cef0-4424-892f-d8ee901bc726\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2e9c2619905eedbbd224c82c2e3dc0934d19acbf355c347fa2f378f811d987e6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3150, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "ref_doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "chroma:document": "# Idefics2\n\n## Overview\n\nThe Idefics2 model was proposed in [What matters when building vision-language models?](https://arxiv.org/abs/2405.02246) by L\u00e9o Tronchon, Hugo Laurencon, Victor Sanh. The accompanying blog post can be found [here](https://huggingface.co/blog/idefics2).\n\nIdefics2 is an open multimodal model that accepts arbitrary sequences of image and text inputs and produces text\noutputs. The model can answer questions about images, describe visual content, create stories grounded on multiple\nimages, or simply behave as a pure language model without visual inputs. It improves upon IDEFICS-1, notably on\ndocument understanding, OCR, or visual reasoning. Idefics2 is lightweight (8 billion parameters) and treats\nimages in their native aspect ratio and resolution, which allows for varying inference efficiency.\n\nThe abstract from the paper is the following:\n\n*The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient foundational VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/idefics2_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Idefics2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2405.02246\">original paper.</a> </small>\n\nThis model was contributed by [amyeroberts](https://huggingface.co/amyeroberts).\nThe original code can be found [here](https://huggingface.co/HuggingFaceM4/idefics2).\n\n## Usage tips\n\n- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images in a batch for input to the model.\n- The processor has a `do_image_splitting` option. If `True`, each input image will be split into 4 sub-images, and concatenated with the original to form 5 images. This is useful for increasing model performance. Make sure `processor.image_processor.do_image_splitting` is set to `False` if the model was not trained with this option.\n- `text` passed to the processor should have the `<image>` tokens where the images should be inserted. And `<end_of_utterance>` at the end of each utterance if the text is a chat message.\n- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as `text` to the processor."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: edf3f144-cef0-4424-892f-d8ee901bc726
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"edf3f144-cef0-4424-892f-d8ee901bc726\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ac6893-6ccd-4c84-a098-a776207f8d3e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b592f40a-ca39-477f-92bb-d8f50519daeb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bff7a27590632959e0b83494b7c36d62c8b61ffa38cdc935b4ef15ccd3d98654\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9f2d3a3c-dadc-400e-bfec-0b5c4c4cb05a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"73f6cfa1b153a27e065e883bbd18df2da0e1c696748e6169faa538c2ce171e27\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1556, \"end_char_idx\": 3813, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "ref_doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "chroma:document": "Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/idefics2_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Idefics2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2405.02246\">original paper.</a> </small>\n\nThis model was contributed by [amyeroberts](https://huggingface.co/amyeroberts).\nThe original code can be found [here](https://huggingface.co/HuggingFaceM4/idefics2).\n\n## Usage tips\n\n- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images in a batch for input to the model.\n- The processor has a `do_image_splitting` option. If `True`, each input image will be split into 4 sub-images, and concatenated with the original to form 5 images. This is useful for increasing model performance. Make sure `processor.image_processor.do_image_splitting` is set to `False` if the model was not trained with this option.\n- `text` passed to the processor should have the `<image>` tokens where the images should be inserted. And `<end_of_utterance>` at the end of each utterance if the text is a chat message.\n- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as `text` to the processor.\n\nExample of how to use the processor on chat messages:\n\n```python\nimport requests\nfrom PIL import Image\nfrom transformers import Idefics2Processor, Idefics2ForConditionalGeneration\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nurl_1 = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nurl_2 = \"http://images.cocodataset.org/val2017/000000219578.jpg\"\n\nimage_1 = Image.open(requests.get(url_1, stream=True).raw)\nimage_2 = Image.open(requests.get(url_2, stream=True).raw)\nimages = [image_1, image_2]\n\nmessages = [{\n    \"role\": \"user\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"What\u2019s the difference between these two images?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9f2d3a3c-dadc-400e-bfec-0b5c4c4cb05a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9f2d3a3c-dadc-400e-bfec-0b5c4c4cb05a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ac6893-6ccd-4c84-a098-a776207f8d3e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"edf3f144-cef0-4424-892f-d8ee901bc726\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0af494fee99b82f9019e53214f58bceacad7a7b1787948798d9cf31232df95ad\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0036b894-2a5a-4ddd-bcc2-8c26f849a4ea\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b3466f9230102948534028cf5a078bdf96b5e6d0f73e2695f6dd430cbc1e5387\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2275, \"end_char_idx\": 4803, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "ref_doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "chroma:document": "## Usage tips\n\n- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images in a batch for input to the model.\n- The processor has a `do_image_splitting` option. If `True`, each input image will be split into 4 sub-images, and concatenated with the original to form 5 images. This is useful for increasing model performance. Make sure `processor.image_processor.do_image_splitting` is set to `False` if the model was not trained with this option.\n- `text` passed to the processor should have the `<image>` tokens where the images should be inserted. And `<end_of_utterance>` at the end of each utterance if the text is a chat message.\n- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as `text` to the processor.\n\nExample of how to use the processor on chat messages:\n\n```python\nimport requests\nfrom PIL import Image\nfrom transformers import Idefics2Processor, Idefics2ForConditionalGeneration\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nurl_1 = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nurl_2 = \"http://images.cocodataset.org/val2017/000000219578.jpg\"\n\nimage_1 = Image.open(requests.get(url_1, stream=True).raw)\nimage_2 = Image.open(requests.get(url_2, stream=True).raw)\nimages = [image_1, image_2]\n\nmessages = [{\n    \"role\": \"user\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"What\u2019s the difference between these two images?\"},\n        {\"type\": \"image\"},\n        {\"type\": \"image\"},\n    ],\n}]\n\nprocessor = Idefics2Processor.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel.to(device)\n\n# at inference time, one needs to pass `add_generation_prompt=True` in order to make sure the model completes the prompt\ntext = processor.apply_chat_template(messages, add_generation_prompt=True)\nprint(text)\n# 'User: What\u2019s the difference between these two images?<image><image><end_of_utterance>\\nAssistant:'\n\ninputs = processor(images=images, text=text, return_tensors=\"pt\").to(device)\n\ngenerated_text = model.generate(**inputs, max_new_tokens=500)\ngenerated_text = processor.batch_decode(generated_text, skip_special_tokens=True)[0]\nprint(\"Generated text:\", generated_text)\n```\n\n- During training, it's important to determine which tokens the model should not learn. For Idefics2, this typically comes down to the image and padding tokens."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0036b894-2a5a-4ddd-bcc2-8c26f849a4ea
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0036b894-2a5a-4ddd-bcc2-8c26f849a4ea\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ac6893-6ccd-4c84-a098-a776207f8d3e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9f2d3a3c-dadc-400e-bfec-0b5c4c4cb05a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e006e2c3a11515a51dc9ca5b47b6a396372159ad50f52f19892e282ef9befb51\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b1e0f372-87dd-4bf4-9cd0-71479e3c68af\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dd497a6a2c212fd436b639beda714e2a78ebf1d80259b40e5c353778051f3a4e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3813, \"end_char_idx\": 6898, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "ref_doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "chroma:document": "\"},\n        {\"type\": \"image\"},\n        {\"type\": \"image\"},\n    ],\n}]\n\nprocessor = Idefics2Processor.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel.to(device)\n\n# at inference time, one needs to pass `add_generation_prompt=True` in order to make sure the model completes the prompt\ntext = processor.apply_chat_template(messages, add_generation_prompt=True)\nprint(text)\n# 'User: What\u2019s the difference between these two images?<image><image><end_of_utterance>\\nAssistant:'\n\ninputs = processor(images=images, text=text, return_tensors=\"pt\").to(device)\n\ngenerated_text = model.generate(**inputs, max_new_tokens=500)\ngenerated_text = processor.batch_decode(generated_text, skip_special_tokens=True)[0]\nprint(\"Generated text:\", generated_text)\n```\n\n- During training, it's important to determine which tokens the model should not learn. For Idefics2, this typically comes down to the image and padding tokens. This means that one can create the labels as follows:\n\n```python\nimport requests\nfrom PIL import Image\nfrom transformers import Idefics2Processor, Idefics2ForConditionalGeneration\nimport torch\n\nurl_1 = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nurl_2 = \"http://images.cocodataset.org/val2017/000000219578.jpg\"\n\nimage_1 = Image.open(requests.get(url_1, stream=True).raw)\nimage_2 = Image.open(requests.get(url_2, stream=True).raw)\nimages = [image_1, image_2]\n\nmessages = [{\n    \"role\": \"user\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"What\u2019s the difference between these two images?\"},\n        {\"type\": \"image\"},\n        {\"type\": \"image\"},\n    ],\n},\n{\n    \"role\": \"assistant\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"The difference is that one image is about dogs and the other one about cats.\"},\n    ],\n}]\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nprocessor = Idefics2Processor.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel.to(device)\n\ntext = processor.apply_chat_template(messages, add_generation_prompt=False)\ninputs = processor(images=images, text=text, return_tensors=\"pt\").to(device)\n\nlabels = inputs.input_ids.clone()\nlabels[labels == processor.tokenizer.pad_token_id] = -100\nlabels[labels == model.config.image_token_id] = -100\n\ninputs[\"labels\"] = labels\n\noutputs = model(**inputs)\nloss = outputs.loss\nloss.backward()\n```\n\nDo note that when training Idefics2 on multi-turn conversations between a user and an assistant, one typically also sets all the tokens corresponding to the user messages to -100.\n\n## Model optimizations: Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b1e0f372-87dd-4bf4-9cd0-71479e3c68af
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b1e0f372-87dd-4bf4-9cd0-71479e3c68af\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ac6893-6ccd-4c84-a098-a776207f8d3e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0036b894-2a5a-4ddd-bcc2-8c26f849a4ea\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"641cb4a9674eb91a0d6d44638d710738be447306dff22ee609c0df298e5953b7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f4cd595f-1b09-4b9f-a742-7a1c0376ff39\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"68725f6522432aad0a3c718574d3f68662557193856c8a429b8f6703178490a3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5408, \"end_char_idx\": 8115, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "ref_doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "chroma:document": "\"},\n        {\"type\": \"image\"},\n        {\"type\": \"image\"},\n    ],\n},\n{\n    \"role\": \"assistant\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"The difference is that one image is about dogs and the other one about cats.\"},\n    ],\n}]\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nprocessor = Idefics2Processor.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel.to(device)\n\ntext = processor.apply_chat_template(messages, add_generation_prompt=False)\ninputs = processor(images=images, text=text, return_tensors=\"pt\").to(device)\n\nlabels = inputs.input_ids.clone()\nlabels[labels == processor.tokenizer.pad_token_id] = -100\nlabels[labels == model.config.image_token_id] = -100\n\ninputs[\"labels\"] = labels\n\noutputs = model(**inputs)\nloss = outputs.loss\nloss.backward()\n```\n\nDo note that when training Idefics2 on multi-turn conversations between a user and an assistant, one typically also sets all the tokens corresponding to the user messages to -100.\n\n## Model optimizations: Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\nTo load and run a model using Flash Attention-2, simply change the code snippet above with the following change:\n\n```diff\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\n    \"HuggingFaceM4/idefics2-8b\",\n+    torch_dtype=torch.float16,    \n+    attn_implementation=\"flash_attention_2\",\n).to(device)\n```\n\n## Shrinking down Idefics2 using quantization\n\nAs the Idefics2 model has 8 billion parameters, that would require about 16GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter), that requires only about 3.5GB of RAM.\n\nQuantizing a model is as simple as passing a `quantization_config` to the model. One can change the code snippet above with the changes below."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f4cd595f-1b09-4b9f-a742-7a1c0376ff39
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f4cd595f-1b09-4b9f-a742-7a1c0376ff39\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ac6893-6ccd-4c84-a098-a776207f8d3e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b1e0f372-87dd-4bf4-9cd0-71479e3c68af\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9d7796fca3eb9d524e4d6d88368c242ee07dd20d4978e6978fd82cdb8bd14456\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"001f10e0-c96f-4707-8b0c-af3846c6b0f4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"401b4134d593065e28611712b0303a7db7d356b3171682ca513cddf3b2048875\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6456, \"end_char_idx\": 9464, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "ref_doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "chroma:document": "## Model optimizations: Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\nTo load and run a model using Flash Attention-2, simply change the code snippet above with the following change:\n\n```diff\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\n    \"HuggingFaceM4/idefics2-8b\",\n+    torch_dtype=torch.float16,    \n+    attn_implementation=\"flash_attention_2\",\n).to(device)\n```\n\n## Shrinking down Idefics2 using quantization\n\nAs the Idefics2 model has 8 billion parameters, that would require about 16GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter), that requires only about 3.5GB of RAM.\n\nQuantizing a model is as simple as passing a `quantization_config` to the model. One can change the code snippet above with the changes below. We'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):\n\n```diff\n+ from transformers import BitsAndBytesConfig\n\n+ quantization_config = BitsAndBytesConfig(\n+    load_in_4bit=True,\n+    bnb_4bit_quant_type=\"nf4\",\n+    bnb_4bit_use_double_quant=True,\n+    bnb_4bit_compute_dtype=torch.float16\n+ )\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\n    \"HuggingFaceM4/idefics2-8b\",\n+    torch_dtype=torch.float16,    \n+    quantization_config=quantization_config,\n).to(device)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Idefics2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A notebook on how to fine-tune Idefics2 on a custom dataset using the [Trainer](../main_classes/trainer.md) can be found [here](https://colab.research.google.com/drive/1NtcTgRbSBKN7pYD3Vdx1j9m8pt3fhFDB?usp=sharing). It supports both full fine-tuning as well as (quantized) LoRa.\n- A script regarding how to fine-tune Idefics2 using the TRL library can be found [here](https://gist.github.com/edbeeching/228652fc6c2b29a1641be5a5778223cb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 001f10e0-c96f-4707-8b0c-af3846c6b0f4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"001f10e0-c96f-4707-8b0c-af3846c6b0f4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ac6893-6ccd-4c84-a098-a776207f8d3e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f4cd595f-1b09-4b9f-a742-7a1c0376ff39\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"71ef05b16e9ec2e8a9c83a278ea4bac3cbd772a2c68b27e8aed1fb401e1331ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8054, \"end_char_idx\": 9992, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "ref_doc_id": "59ac6893-6ccd-4c84-a098-a776207f8d3e", "chroma:document": "One can change the code snippet above with the changes below. We'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):\n\n```diff\n+ from transformers import BitsAndBytesConfig\n\n+ quantization_config = BitsAndBytesConfig(\n+    load_in_4bit=True,\n+    bnb_4bit_quant_type=\"nf4\",\n+    bnb_4bit_use_double_quant=True,\n+    bnb_4bit_compute_dtype=torch.float16\n+ )\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\n    \"HuggingFaceM4/idefics2-8b\",\n+    torch_dtype=torch.float16,    \n+    quantization_config=quantization_config,\n).to(device)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Idefics2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A notebook on how to fine-tune Idefics2 on a custom dataset using the [Trainer](../main_classes/trainer.md) can be found [here](https://colab.research.google.com/drive/1NtcTgRbSBKN7pYD3Vdx1j9m8pt3fhFDB?usp=sharing). It supports both full fine-tuning as well as (quantized) LoRa.\n- A script regarding how to fine-tune Idefics2 using the TRL library can be found [here](https://gist.github.com/edbeeching/228652fc6c2b29a1641be5a5778223cb).\n- Demo notebook regarding fine-tuning Idefics2 for JSON extraction use cases can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Idefics2). \ud83c\udf0e\n\n## Idefics2Config\n\n[[autodoc]] Idefics2Config\n\n\n## Idefics2Model\n\n[[autodoc]] Idefics2Model\n    - forward\n\n\n## Idefics2ForConditionalGeneration\n\n[[autodoc]] Idefics2ForConditionalGeneration\n    - forward\n\n\n## Idefics2ImageProcessor\n[[autodoc]] Idefics2ImageProcessor\n    - preprocess\n\n\n## Idefics2Processor\n[[autodoc]] Idefics2Processor\n    - __call__"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5b6315fc-c47f-41d8-9259-0a6ab6c9ea60
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/informer", "title": "Informer", "tokens": 782, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5b6315fc-c47f-41d8-9259-0a6ab6c9ea60\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/informer\", \"title\": \"Informer\", \"tokens\": 782, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"743d9f70-8a57-427f-8825-add1494a86d0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/informer\", \"title\": \"Informer\", \"tokens\": 782, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"419433c4d6cda15872e3e0ae618b41016541dd044f257b8d02cb46167168c180\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2966, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "743d9f70-8a57-427f-8825-add1494a86d0", "doc_id": "743d9f70-8a57-427f-8825-add1494a86d0", "ref_doc_id": "743d9f70-8a57-427f-8825-add1494a86d0", "chroma:document": "# Informer\n\n## Overview\n\nThe Informer model was proposed in [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://arxiv.org/abs/2012.07436) by Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.\n\nThis method introduces a Probabilistic Attention mechanism to select the \"active\" queries rather than the \"lazy\" queries and provides a sparse Transformer thus mitigating the quadratic compute and memory requirements of vanilla attention.\n\nThe abstract from the paper is the following:\n\n*Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves O(L logL) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.*\n\nThis model was contributed by [elisim](https://huggingface.co/elisim) and [kashif](https://huggingface.co/kashif).\nThe original code can be found [here](https://github.com/zhouhaoyi/Informer2020).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Check out the Informer blog-post in HuggingFace blog: [Multivariate Probabilistic Time Series Forecasting with Informer](https://huggingface.co/blog/informer)\n\n## InformerConfig\n\n[[autodoc]] InformerConfig\n\n## InformerModel\n\n[[autodoc]] InformerModel\n    - forward\n\n## InformerForPrediction\n\n[[autodoc]] InformerForPrediction\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 09a96459-5445-41c2-9275-65969d2604f7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/herbert", "title": "HerBERT", "tokens": 804, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"09a96459-5445-41c2-9275-65969d2604f7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/herbert\", \"title\": \"HerBERT\", \"tokens\": 804, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"baa6b46b-9237-43f5-aa29-9272c512869b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/herbert\", \"title\": \"HerBERT\", \"tokens\": 804, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c32ff69b7c80d4f1b0968c4afa3e0693e6a59c2f73ac35a314e05eb40bc7f1b4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2839, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "baa6b46b-9237-43f5-aa29-9272c512869b", "doc_id": "baa6b46b-9237-43f5-aa29-9272c512869b", "ref_doc_id": "baa6b46b-9237-43f5-aa29-9272c512869b", "chroma:document": "# HerBERT\n\n## Overview\n\nThe HerBERT model was proposed in [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://www.aclweb.org/anthology/2020.acl-main.111.pdf) by Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and\nIreneusz Gawlik. It is a BERT-based Language Model trained on Polish Corpora using only MLM objective with dynamic\nmasking of whole words.\n\nThe abstract from the paper is the following:\n\n*In recent years, a series of Transformer-based models unlocked major improvements in general natural language\nunderstanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which\nallow for a fair comparison of the proposed methods. However, such benchmarks are available only for a handful of\nlanguages. To alleviate this issue, we introduce a comprehensive multi-task benchmark for the Polish language\nunderstanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing\ndatasets for named entity recognition, question-answering, textual entailment, and others. We also introduce a new\nsentiment analysis task for the e-commerce domain, named Allegro Reviews (AR). To ensure a common evaluation scheme and\npromote models that generalize to different NLU tasks, the benchmark includes datasets from varying domains and\napplications. Additionally, we release HerBERT, a Transformer-based model trained specifically for the Polish language,\nwhich has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an\nextensive evaluation, including several standard baselines and recently proposed, multilingual Transformer-based\nmodels.*\n\nThis model was contributed by [rmroczkowski](https://huggingface.co/rmroczkowski). The original code can be found\n[here](https://github.com/allegro/HerBERT).\n\n\n## Usage example\n\n```python\n>>> from transformers import HerbertTokenizer, RobertaModel\n\n>>> tokenizer = HerbertTokenizer.from_pretrained(\"allegro/herbert-klej-cased-tokenizer-v1\")\n>>> model = RobertaModel.from_pretrained(\"allegro/herbert-klej-cased-v1\")\n\n>>> encoded_input = tokenizer.encode(\"Kto ma lepsz\u0105 sztuk\u0119, ma lepszy rz\u0105d \u2013 to jasne.\", return_tensors=\"pt\")\n>>> outputs = model(encoded_input)\n\n>>> # HerBERT can also be loaded using AutoTokenizer and AutoModel:\n>>> import torch\n>>> from transformers import AutoModel, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"allegro/herbert-klej-cased-tokenizer-v1\")\n>>> model = AutoModel.from_pretrained(\"allegro/herbert-klej-cased-v1\")\n```\n\n<Tip>\n\nHerbert implementation is the same as `BERT` except for the tokenization method. Refer to [BERT documentation](bert) \nfor API reference and examples.  \n\n</Tip>\n\n## HerbertTokenizer\n\n[[autodoc]] HerbertTokenizer\n\n## HerbertTokenizerFast\n\n[[autodoc]] HerbertTokenizerFast"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 728f32ee-1c35-43e6-a316-435a541a383d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encodec", "title": "EnCodec", "tokens": 859, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"728f32ee-1c35-43e6-a316-435a541a383d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encodec\", \"title\": \"EnCodec\", \"tokens\": 859, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"889efd82-1b53-4870-a817-707c3c33da5f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encodec\", \"title\": \"EnCodec\", \"tokens\": 859, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a22ef4613ac308a3e7d00b0c01fd91f05bbc7d2ab96a8e1326505474d543759d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3127, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "889efd82-1b53-4870-a817-707c3c33da5f", "doc_id": "889efd82-1b53-4870-a817-707c3c33da5f", "ref_doc_id": "889efd82-1b53-4870-a817-707c3c33da5f", "chroma:document": "# EnCodec\n\n## Overview\n\nThe EnCodec neural codec model was proposed in [High Fidelity Neural Audio Compression](https://arxiv.org/abs/2210.13438) by Alexandre D\u00e9fossez, Jade Copet, Gabriel Synnaeve, Yossi Adi.\n\nThe abstract from the paper is the following:\n\n*We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio.*\n\nThis model was contributed by [Matthijs](https://huggingface.co/Matthijs), [Patrick Von Platen](https://huggingface.co/patrickvonplaten) and [Arthur Zucker](https://huggingface.co/ArthurZ). \nThe original code can be found [here](https://github.com/facebookresearch/encodec).\n\n## Usage example \n\nHere is a quick example of how to encode and decode an audio using this model:\n\n```python \n>>> from datasets import load_dataset, Audio\n>>> from transformers import EncodecModel, AutoProcessor\n>>> librispeech_dummy = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n\n>>> model = EncodecModel.from_pretrained(\"facebook/encodec_24khz\")\n>>> processor = AutoProcessor.from_pretrained(\"facebook/encodec_24khz\")\n>>> librispeech_dummy = librispeech_dummy.cast_column(\"audio\", Audio(sampling_rate=processor.sampling_rate))\n>>> audio_sample = librispeech_dummy[-1][\"audio\"][\"array\"]\n>>> inputs = processor(raw_audio=audio_sample, sampling_rate=processor.sampling_rate, return_tensors=\"pt\")\n\n>>> encoder_outputs = model.encode(inputs[\"input_values\"], inputs[\"padding_mask\"])\n>>> audio_values = model.decode(encoder_outputs.audio_codes, encoder_outputs.audio_scales, inputs[\"padding_mask\"])[0]\n>>> # or the equivalent with a forward pass\n>>> audio_values = model(inputs[\"input_values\"], inputs[\"padding_mask\"]).audio_values\n```\n\n## EncodecConfig\n\n[[autodoc]] EncodecConfig\n\n## EncodecFeatureExtractor\n\n[[autodoc]] EncodecFeatureExtractor\n    - __call__\n\n## EncodecModel\n\n[[autodoc]] EncodecModel\n    - decode\n    - encode\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3643841f-0b79-43bc-b0f8-cdf5927893bc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blip-2", "title": "BLIP-2", "tokens": 1084, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3643841f-0b79-43bc-b0f8-cdf5927893bc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"04e5011a-d14f-471c-8355-67d5442fe946\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4bc38ebc73af47e0b749ecfeff29cfedbd374b999759ea0a908c3ef2d061dabe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ca5a002a-5e42-4384-8e4e-45f97cd8bb74\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ccfce4e31e27ed8ce348277caa67d75365ca27819f56425c13f094d5110ccba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3084, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "04e5011a-d14f-471c-8355-67d5442fe946", "doc_id": "04e5011a-d14f-471c-8355-67d5442fe946", "ref_doc_id": "04e5011a-d14f-471c-8355-67d5442fe946", "chroma:document": "# BLIP-2\n\n## Overview\n\nThe BLIP-2 model was proposed in [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597) by\nJunnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. BLIP-2 leverages frozen pre-trained image encoders and large language models (LLMs) by training a lightweight, 12-layer Transformer\nencoder in between them, achieving state-of-the-art performance on various vision-language tasks. Most notably, BLIP-2 improves upon [Flamingo](https://arxiv.org/abs/2204.14198), an 80 billion parameter model, by 8.7%\non zero-shot VQAv2 with 54x fewer trainable parameters. \n\nThe abstract from the paper is the following:\n\n*The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> BLIP-2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2301.12597\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/salesforce/LAVIS/tree/5ee63d688ba4cebff63acee04adaef2dee9af207).\n\n## Usage tips\n\n- BLIP-2 can be used for conditional text generation given an image and an optional text prompt. At inference time, it's recommended to use the [`generate`] method.\n- One can use [`Blip2Processor`] to prepare images for the model, and decode the predicted tokens ID's back to text.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BLIP-2.\n\n- Demo notebooks for BLIP-2 for image captioning, visual question answering (VQA) and chat-like conversations can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BLIP-2).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ca5a002a-5e42-4384-8e4e-45f97cd8bb74
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blip-2", "title": "BLIP-2", "tokens": 1084, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ca5a002a-5e42-4384-8e4e-45f97cd8bb74\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"04e5011a-d14f-471c-8355-67d5442fe946\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4bc38ebc73af47e0b749ecfeff29cfedbd374b999759ea0a908c3ef2d061dabe\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3643841f-0b79-43bc-b0f8-cdf5927893bc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b04e5cd283e4bf147f11735d54224805a17fcc866aab881b4b8a7dbecf15c456\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1502, \"end_char_idx\": 3686, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "04e5011a-d14f-471c-8355-67d5442fe946", "doc_id": "04e5011a-d14f-471c-8355-67d5442fe946", "ref_doc_id": "04e5011a-d14f-471c-8355-67d5442fe946", "chroma:document": "For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> BLIP-2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2301.12597\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/salesforce/LAVIS/tree/5ee63d688ba4cebff63acee04adaef2dee9af207).\n\n## Usage tips\n\n- BLIP-2 can be used for conditional text generation given an image and an optional text prompt. At inference time, it's recommended to use the [`generate`] method.\n- One can use [`Blip2Processor`] to prepare images for the model, and decode the predicted tokens ID's back to text.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BLIP-2.\n\n- Demo notebooks for BLIP-2 for image captioning, visual question answering (VQA) and chat-like conversations can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BLIP-2).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## Blip2Config\n\n[[autodoc]] Blip2Config\n    - from_vision_qformer_text_configs\n\n## Blip2VisionConfig\n\n[[autodoc]] Blip2VisionConfig\n\n## Blip2QFormerConfig\n\n[[autodoc]] Blip2QFormerConfig\n\n## Blip2Processor\n\n[[autodoc]] Blip2Processor\n\n## Blip2VisionModel\n\n[[autodoc]] Blip2VisionModel\n    - forward\n\n## Blip2QFormerModel\n\n[[autodoc]] Blip2QFormerModel\n    - forward\n\n## Blip2Model\n\n[[autodoc]] Blip2Model\n    - forward\n    - get_text_features\n    - get_image_features\n    - get_qformer_features\n\n## Blip2ForConditionalGeneration\n\n[[autodoc]] Blip2ForConditionalGeneration\n    - forward\n    - generate"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 18606f70-f28a-42a7-b104-06ec7e87ceb6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"18606f70-f28a-42a7-b104-06ec7e87ceb6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7feac494-1b6d-47ad-a648-da292ea74b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5fb7df1e-0aef-41c4-a55d-590805e82e85\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"88eaddb12c295fb028a6679b546f5d8db9e7856402de575dc51d1a2bc812b315\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3295, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "ref_doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "chroma:document": "# BERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=bert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-bert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/bert-base-uncased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe BERT model was proposed in [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. It's a\nbidirectional transformer pretrained using a combination of masked language modeling objective and next sentence\nprediction on a large corpus comprising the Toronto Book Corpus and Wikipedia.\n\nThe abstract from the paper is the following:\n\n*We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations\nfrom Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional\nrepresentations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result,\nthe pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models\nfor a wide range of tasks, such as question answering and language inference, without substantial task-specific\narchitecture modifications.*\n\n*BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural\nlanguage processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI\naccuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute\nimprovement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/bert).\n\n## Usage tips\n\n- BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- BERT was trained with the masked language modeling (MLM) and next sentence prediction (NSP) objectives. It is\n  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation.\n- Corrupts the inputs by using random masking, more precisely, during pretraining, a given percentage of tokens (usually 15%) is masked by:\n\n    * a special mask token with probability 0.8\n    * a random token different from the one masked with probability 0.1\n    * the same token with probability 0.1\n    \n- The model must predict the original sentence, but has a second objective: inputs are two sentences A and B (with a separation token in between). With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5fb7df1e-0aef-41c4-a55d-590805e82e85
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5fb7df1e-0aef-41c4-a55d-590805e82e85\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7feac494-1b6d-47ad-a648-da292ea74b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"18606f70-f28a-42a7-b104-06ec7e87ceb6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"355590af18a9342600aae8a8e1c141a8cbe5045324a37fb38a590ba1ff98bccb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"81ba3efd-9127-4c3b-a1bd-1b0bea0d45f4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"56260d25965577b9debcd773320033151e7c2d60124058c3f914b3dc667d34f8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1872, \"end_char_idx\": 4217, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "ref_doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "chroma:document": "*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/bert).\n\n## Usage tips\n\n- BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- BERT was trained with the masked language modeling (MLM) and next sentence prediction (NSP) objectives. It is\n  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation.\n- Corrupts the inputs by using random masking, more precisely, during pretraining, a given percentage of tokens (usually 15%) is masked by:\n\n    * a special mask token with probability 0.8\n    * a random token different from the one masked with probability 0.1\n    * the same token with probability 0.1\n    \n- The model must predict the original sentence, but has a second objective: inputs are two sentences A and B (with a separation token in between). With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import BertModel\n\nmodel = BertModel.from_pretrained(\"bert-base-uncased\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-80GB, CPUx12, RAM 96.6GB, PyTorch 2.2.0, OS Ubuntu 22.04) with `float16`, we saw the \nfollowing speedups during training and inference."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 81ba3efd-9127-4c3b-a1bd-1b0bea0d45f4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"81ba3efd-9127-4c3b-a1bd-1b0bea0d45f4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7feac494-1b6d-47ad-a648-da292ea74b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5fb7df1e-0aef-41c4-a55d-590805e82e85\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"dd16fc2dff1e5010339f48a6aa11d6ff17a730c28d9bfa157d5023848c30abeb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"70fbd2b6-3c24-4bf0-a480-1d4c8bdc92c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c69cf33c2f8a695fe3d2039c85992cb4141057bb08c52c9070e5ed3d540d4e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2847, \"end_char_idx\": 6418, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "ref_doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "chroma:document": "With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import BertModel\n\nmodel = BertModel.from_pretrained(\"bert-base-uncased\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-80GB, CPUx12, RAM 96.6GB, PyTorch 2.2.0, OS Ubuntu 22.04) with `float16`, we saw the \nfollowing speedups during training and inference.\n\n#### Training\n\n|batch_size|seq_len|Time per batch (eager - s)|Time per batch (sdpa - s)|Speedup (%)|Eager peak mem (MB)|sdpa peak mem (MB)|Mem saving (%)|\n|----------|-------|--------------------------|-------------------------|-----------|-------------------|------------------|--------------|\n|4         |256    |0.023                     |0.017                    |35.472     |939.213            |764.834           |22.800        |\n|4         |512    |0.023                     |0.018                    |23.687     |1970.447           |1227.162          |60.569        |\n|8         |256    |0.023                     |0.018                    |23.491     |1594.295           |1226.114          |30.028        |\n|8         |512    |0.035                     |0.025                    |43.058     |3629.401           |2134.262          |70.054        |\n|16        |256    |0.030                     |0.024                    |25.583     |2874.426           |2134.262          |34.680        |\n|16        |512    |0.064                     |0.044                    |46.223     |6964.659           |3961.013          |75.830        |\n\n#### Inference\n\n|batch_size|seq_len|Per token latency eager (ms)|Per token latency SDPA (ms)|Speedup (%)|Mem eager (MB)|Mem BT (MB)|Mem saved (%)|\n|----------|-------|----------------------------|---------------------------|-----------|--------------|-----------|-------------|\n|1         |128    |5.736                       |4.987                      |15.022     |282.661       |282.924    |-0.093       |\n|1         |256    |5.689                       |4.945                      |15.055     |298.686       |298.948    |-0.088       |\n|2         |128    |6.154                       |4.982                      |23.521     |314.523       |314.785    |-0.083       |\n|2         |256    |6.201                       |4.949                      |25.303     |347.546       |347.033    |0.148        |\n|4         |128    |6.049                       |4.987                      |21.305     |378.895       |379.301    |-0.107       |\n|4         |256    |6.285                       |5.364                      |17.166     |443.209       |444.382    |-0.264       |"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 70fbd2b6-3c24-4bf0-a480-1d4c8bdc92c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"70fbd2b6-3c24-4bf0-a480-1d4c8bdc92c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7feac494-1b6d-47ad-a648-da292ea74b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"81ba3efd-9127-4c3b-a1bd-1b0bea0d45f4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"64ee383e73ceaa65fbf9084bb0674354aad45b1bd4e359166d5d9ece600d75b0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b10012c8-bf97-49ea-9b47-27768e4d921e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"def6929d6cb1a9aa7ee8e3c4dd1570112e269ee633a6aeea9ca0681cd47bc9f3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6422, \"end_char_idx\": 9651, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "ref_doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "chroma:document": "## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [BERT Text Classification in a different language](https://www.philschmid.de/bert-text-classification-in-a-different-language).\n- A notebook for [Finetuning BERT (and friends) for multi-label text classification](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb).\n- A notebook on how to [Finetune BERT for multi-label classification using PyTorch](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb). \ud83c\udf0e\n- A notebook on how to [warm-start an EncoderDecoder model with BERT for summarization](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb).\n- [`BertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).\n- [`FlaxBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A blog post on how to use [Hugging Face Transformers with Keras: Fine-tune a non-English BERT for Named Entity Recognition](https://www.philschmid.de/huggingface-transformers-keras-tf).\n- A notebook for [Finetuning BERT for named-entity recognition](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT_only_first_wordpiece.ipynb) using only the first wordpiece of each word in the word label during tokenization. To propagate the label of the word to all wordpieces, see this [version](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT.ipynb) of the notebook instead.\n- [`BertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b10012c8-bf97-49ea-9b47-27768e4d921e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b10012c8-bf97-49ea-9b47-27768e4d921e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7feac494-1b6d-47ad-a648-da292ea74b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"70fbd2b6-3c24-4bf0-a480-1d4c8bdc92c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c49ecad549e83593a46b37e86e77ffc09dd1e0162eada8b984c52010676b33d2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"883c481a-aa9e-4675-be19-7e0c654ff80a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e299b0ad534f982b3b59f1e84cbb864c29c47248df5ffc5095462e6c0bc0f70a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8233, \"end_char_idx\": 11372, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "ref_doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "chroma:document": "- [`FlaxBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A blog post on how to use [Hugging Face Transformers with Keras: Fine-tune a non-English BERT for Named Entity Recognition](https://www.philschmid.de/huggingface-transformers-keras-tf).\n- A notebook for [Finetuning BERT for named-entity recognition](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT_only_first_wordpiece.ipynb) using only the first wordpiece of each word in the word label during tokenization. To propagate the label of the word to all wordpieces, see this [version](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT.ipynb) of the notebook instead.\n- [`BertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).\n- [`TFBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [`FlaxBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`BertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 883c481a-aa9e-4675-be19-7e0c654ff80a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"883c481a-aa9e-4675-be19-7e0c654ff80a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7feac494-1b6d-47ad-a648-da292ea74b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b10012c8-bf97-49ea-9b47-27768e4d921e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"865458efce7d4e9bf8fa295d231d7a5aa4f27a3ffea7d962726d81646b445bb3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e4efb7f0-687b-411c-8b6d-7b05eca09b95\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"256a523c841795735baf57a8e6e33e7916b89edf7ca973b48a589892b04ddd3a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9950, \"end_char_idx\": 12983, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "ref_doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "chroma:document": "- [`FlaxBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`BertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`BertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n- [`BertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFBertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e4efb7f0-687b-411c-8b6d-7b05eca09b95
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e4efb7f0-687b-411c-8b6d-7b05eca09b95\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7feac494-1b6d-47ad-a648-da292ea74b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"883c481a-aa9e-4675-be19-7e0c654ff80a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"260306f9fb9c0b1d32b6c5aa8a2b72bae5c1adb775c80d30c5598ad060b26e4b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0184e500-50f8-4f0a-b406-dd29f52622ff\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f2a867e9151f9196b3812d1ed74925fbe0495840bdfaee09ab74281f1e07f7b7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11373, \"end_char_idx\": 14364, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "ref_doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "chroma:document": "- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`BertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n- [`BertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFBertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n\u26a1\ufe0f **Inference**\n- A blog post on how to [Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia](https://huggingface.co/blog/bert-inferentia-sagemaker).\n- A blog post on how to [Accelerate BERT inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/bert-deepspeed-inference).\n\n\u2699\ufe0f **Pretraining**\n- A blog post on [Pre-Training BERT with Hugging Face Transformers and Habana Gaudi](https://www.philschmid.de/pre-training-bert-habana).\n\n\ud83d\ude80 **Deploy**\n- A blog post on how to [Convert Transformers to ONNX with Hugging Face Optimum](https://www.philschmid.de/convert-transformers-to-onnx).\n- A blog post on how to [Setup Deep Learning environment for Hugging Face Transformers with Habana Gaudi on AWS](https://www.philschmid.de/getting-started-habana-gaudi#conclusion).\n- A blog post on [Autoscaling BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module](https://www.philschmid.de/terraform-huggingface-amazon-sagemaker-advanced).\n- A blog post on [Serverless BERT with HuggingFace, AWS Lambda, and Docker](https://www.philschmid.de/serverless-bert-with-huggingface-aws-lambda-docker).\n- A blog post on [Hugging Face Transformers BERT fine-tuning using Amazon SageMaker and Training Compiler](https://www.philschmid.de/huggingface-amazon-sagemaker-training-compiler)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0184e500-50f8-4f0a-b406-dd29f52622ff
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0184e500-50f8-4f0a-b406-dd29f52622ff\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7feac494-1b6d-47ad-a648-da292ea74b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4efb7f0-687b-411c-8b6d-7b05eca09b95\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"81fe5ef6e13de34f6b0c2379bd66542d875f418cc02cd0cafc43d35ffc2e3d94\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"85c2e9f8-e5ce-4fac-a37a-e63e23414435\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"63e76fbef7d093557edd0f166ffcd4634c9960737c7883277aaf86bb345b81ee\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12984, \"end_char_idx\": 15162, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "ref_doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "chroma:document": "- [Multiple choice task guide](../tasks/multiple_choice)\n\n\u26a1\ufe0f **Inference**\n- A blog post on how to [Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia](https://huggingface.co/blog/bert-inferentia-sagemaker).\n- A blog post on how to [Accelerate BERT inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/bert-deepspeed-inference).\n\n\u2699\ufe0f **Pretraining**\n- A blog post on [Pre-Training BERT with Hugging Face Transformers and Habana Gaudi](https://www.philschmid.de/pre-training-bert-habana).\n\n\ud83d\ude80 **Deploy**\n- A blog post on how to [Convert Transformers to ONNX with Hugging Face Optimum](https://www.philschmid.de/convert-transformers-to-onnx).\n- A blog post on how to [Setup Deep Learning environment for Hugging Face Transformers with Habana Gaudi on AWS](https://www.philschmid.de/getting-started-habana-gaudi#conclusion).\n- A blog post on [Autoscaling BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module](https://www.philschmid.de/terraform-huggingface-amazon-sagemaker-advanced).\n- A blog post on [Serverless BERT with HuggingFace, AWS Lambda, and Docker](https://www.philschmid.de/serverless-bert-with-huggingface-aws-lambda-docker).\n- A blog post on [Hugging Face Transformers BERT fine-tuning using Amazon SageMaker and Training Compiler](https://www.philschmid.de/huggingface-amazon-sagemaker-training-compiler).\n- A blog post on [Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker](https://www.philschmid.de/knowledge-distillation-bert-transformers).\n\n## BertConfig\n\n[[autodoc]] BertConfig\n    - all\n\n## BertTokenizer\n\n[[autodoc]] BertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## BertTokenizerFast\n\n[[autodoc]] BertTokenizerFast\n\n</pt>\n<tf>\n\n## TFBertTokenizer\n\n[[autodoc]] TFBertTokenizer\n\n</tf>\n</frameworkcontent>\n\n## Bert specific outputs\n\n[[autodoc]] models.bert.modeling_bert.BertForPreTrainingOutput\n\n[[autodoc]] models.bert.modeling_tf_bert.TFBertForPreTrainingOutput\n\n[[autodoc]] models.bert.modeling_flax_bert.FlaxBertForPreTrainingOutput"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 85c2e9f8-e5ce-4fac-a37a-e63e23414435
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"85c2e9f8-e5ce-4fac-a37a-e63e23414435\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7feac494-1b6d-47ad-a648-da292ea74b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0184e500-50f8-4f0a-b406-dd29f52622ff\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c46e81f3fe5a65c7d062d659bea0f067e31a0b19931402a2d06aeeab73d8aa9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13665, \"end_char_idx\": 17343, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "ref_doc_id": "7feac494-1b6d-47ad-a648-da292ea74b82", "chroma:document": "- A blog post on how to [Setup Deep Learning environment for Hugging Face Transformers with Habana Gaudi on AWS](https://www.philschmid.de/getting-started-habana-gaudi#conclusion).\n- A blog post on [Autoscaling BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module](https://www.philschmid.de/terraform-huggingface-amazon-sagemaker-advanced).\n- A blog post on [Serverless BERT with HuggingFace, AWS Lambda, and Docker](https://www.philschmid.de/serverless-bert-with-huggingface-aws-lambda-docker).\n- A blog post on [Hugging Face Transformers BERT fine-tuning using Amazon SageMaker and Training Compiler](https://www.philschmid.de/huggingface-amazon-sagemaker-training-compiler).\n- A blog post on [Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker](https://www.philschmid.de/knowledge-distillation-bert-transformers).\n\n## BertConfig\n\n[[autodoc]] BertConfig\n    - all\n\n## BertTokenizer\n\n[[autodoc]] BertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## BertTokenizerFast\n\n[[autodoc]] BertTokenizerFast\n\n</pt>\n<tf>\n\n## TFBertTokenizer\n\n[[autodoc]] TFBertTokenizer\n\n</tf>\n</frameworkcontent>\n\n## Bert specific outputs\n\n[[autodoc]] models.bert.modeling_bert.BertForPreTrainingOutput\n\n[[autodoc]] models.bert.modeling_tf_bert.TFBertForPreTrainingOutput\n\n[[autodoc]] models.bert.modeling_flax_bert.FlaxBertForPreTrainingOutput\n\n\n<frameworkcontent>\n<pt>\n\n## BertModel\n\n[[autodoc]] BertModel\n    - forward\n\n## BertForPreTraining\n\n[[autodoc]] BertForPreTraining\n    - forward\n\n## BertLMHeadModel\n\n[[autodoc]] BertLMHeadModel\n    - forward\n\n## BertForMaskedLM\n\n[[autodoc]] BertForMaskedLM\n    - forward\n\n## BertForNextSentencePrediction\n\n[[autodoc]] BertForNextSentencePrediction\n    - forward\n\n## BertForSequenceClassification\n\n[[autodoc]] BertForSequenceClassification\n    - forward\n\n## BertForMultipleChoice\n\n[[autodoc]] BertForMultipleChoice\n    - forward\n\n## BertForTokenClassification\n\n[[autodoc]] BertForTokenClassification\n    - forward\n\n## BertForQuestionAnswering\n\n[[autodoc]] BertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFBertModel\n\n[[autodoc]] TFBertModel\n    - call\n\n## TFBertForPreTraining\n\n[[autodoc]] TFBertForPreTraining\n    - call\n\n## TFBertModelLMHeadModel\n\n[[autodoc]] TFBertLMHeadModel\n    - call\n\n## TFBertForMaskedLM\n\n[[autodoc]] TFBertForMaskedLM\n    - call\n\n## TFBertForNextSentencePrediction\n\n[[autodoc]] TFBertForNextSentencePrediction\n    - call\n\n## TFBertForSequenceClassification\n\n[[autodoc]] TFBertForSequenceClassification\n    - call\n\n## TFBertForMultipleChoice\n\n[[autodoc]] TFBertForMultipleChoice\n    - call\n\n## TFBertForTokenClassification\n\n[[autodoc]] TFBertForTokenClassification\n    - call\n\n## TFBertForQuestionAnswering\n\n[[autodoc]] TFBertForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxBertModel\n\n[[autodoc]] FlaxBertModel\n    - __call__\n\n## FlaxBertForPreTraining\n\n[[autodoc]] FlaxBertForPreTraining\n    - __call__\n\n## FlaxBertForCausalLM\n\n[[autodoc]] FlaxBertForCausalLM\n    - __call__\n\n## FlaxBertForMaskedLM\n\n[[autodoc]] FlaxBertForMaskedLM\n    - __call__\n\n## FlaxBertForNextSentencePrediction\n\n[[autodoc]] FlaxBertForNextSentencePrediction\n    - __call__\n\n## FlaxBertForSequenceClassification\n\n[[autodoc]] FlaxBertForSequenceClassification\n    - __call__\n\n## FlaxBertForMultipleChoice\n\n[[autodoc]] FlaxBertForMultipleChoice\n    - __call__\n\n## FlaxBertForTokenClassification\n\n[[autodoc]] FlaxBertForTokenClassification\n    - __call__\n\n## FlaxBertForQuestionAnswering\n\n[[autodoc]] FlaxBertForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8276d3b5-2043-4e64-b786-c666432c7e05
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/umt5", "title": "UMT5", "tokens": 1296, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8276d3b5-2043-4e64-b786-c666432c7e05\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"35bf33fa-c858-4005-aa18-a5fdb6e92c9b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ac3c31740feb43025018d8163b7f83e604ea0dd81ff24a63df0c4226c1315de8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8de343d7-b512-4ecf-9126-58f75640baab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"27f919211001c19c4b92aef3cf267e3129b2129c077ff4e93e23a15d7107711a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2993, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "35bf33fa-c858-4005-aa18-a5fdb6e92c9b", "doc_id": "35bf33fa-c858-4005-aa18-a5fdb6e92c9b", "ref_doc_id": "35bf33fa-c858-4005-aa18-a5fdb6e92c9b", "chroma:document": "# UMT5\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=umt5\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-mt5-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/mt5-small-finetuned-arxiv-cs-finetuned-arxiv-cs-full\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe UMT5 model was proposed in [UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining](https://openreview.net/forum?id=kXwdL1cWOAi) by Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, Noah Constant.\n\nThe abstract from the paper is the following:\n\n*Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages. However previous work has not systematically evaluated the efficacy of different pretraining language distributions across model scales. In this paper, we propose a new sampling method, UniMax, that delivers more uniform coverage of head languages while mitigating overfitting on tail languages by explicitly capping the number of repeats over each language's corpus. We perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale. We find that UniMax outperforms standard temperature-based sampling, and the benefits persist as scale increases. As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.*\n\nGoogle has released the following variants:\n\n- [google/umt5-small](https://huggingface.co/google/umt5-small)\n- [google/umt5-base](https://huggingface.co/google/umt5-base)\n- [google/umt5-xl](https://huggingface.co/google/umt5-xl)\n- [google/umt5-xxl](https://huggingface.co/google/umt5-xxl).\n\nThis model was contributed by [agemagician](https://huggingface.co/agemagician) and [stefan-it](https://huggingface.co/stefan-it). The original code can be\nfound [here](https://github.com/google-research/t5x).\n\n## Usage tips \n\n- UMT5 was only pre-trained on [mC4](https://huggingface.co/datasets/mc4) excluding any supervised training.\nTherefore, this model has to be fine-tuned before it is usable on a downstream task, unlike the original T5 model.\n- Since umT5 was pre-trained in an unsupervised manner, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\n## Differences with mT5?\n`UmT5` is based on mT5, with a non-shared relative positional bias that is computed for each layer. This means that the model set `has_relative_bias` for each layer.\nThe conversion script is also different because the model was saved in t5x's latest checkpointing format."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8de343d7-b512-4ecf-9126-58f75640baab
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/umt5", "title": "UMT5", "tokens": 1296, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8de343d7-b512-4ecf-9126-58f75640baab\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"35bf33fa-c858-4005-aa18-a5fdb6e92c9b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ac3c31740feb43025018d8163b7f83e604ea0dd81ff24a63df0c4226c1315de8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8276d3b5-2043-4e64-b786-c666432c7e05\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"24232d1743b484c035eac6f9707e030b5b6afe3bbe831c27e337993672846348\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1511, \"end_char_idx\": 4254, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "35bf33fa-c858-4005-aa18-a5fdb6e92c9b", "doc_id": "35bf33fa-c858-4005-aa18-a5fdb6e92c9b", "ref_doc_id": "35bf33fa-c858-4005-aa18-a5fdb6e92c9b", "chroma:document": "As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.*\n\nGoogle has released the following variants:\n\n- [google/umt5-small](https://huggingface.co/google/umt5-small)\n- [google/umt5-base](https://huggingface.co/google/umt5-base)\n- [google/umt5-xl](https://huggingface.co/google/umt5-xl)\n- [google/umt5-xxl](https://huggingface.co/google/umt5-xxl).\n\nThis model was contributed by [agemagician](https://huggingface.co/agemagician) and [stefan-it](https://huggingface.co/stefan-it). The original code can be\nfound [here](https://github.com/google-research/t5x).\n\n## Usage tips \n\n- UMT5 was only pre-trained on [mC4](https://huggingface.co/datasets/mc4) excluding any supervised training.\nTherefore, this model has to be fine-tuned before it is usable on a downstream task, unlike the original T5 model.\n- Since umT5 was pre-trained in an unsupervised manner, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\n## Differences with mT5?\n`UmT5` is based on mT5, with a non-shared relative positional bias that is computed for each layer. This means that the model set `has_relative_bias` for each layer.\nThe conversion script is also different because the model was saved in t5x's latest checkpointing format.\n\n# Sample usage\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/umt5-small\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/umt5-small\")\n\n>>> inputs = tokenizer(\n...     \"A <extra_id_0> walks into a bar and orders a <extra_id_1> with <extra_id_2> pinch of <extra_id_3>.\",\n...     return_tensors=\"pt\",\n... )\n>>> outputs = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(outputs))\n['<pad><extra_id_0>nyone who<extra_id_1> drink<extra_id_2> a<extra_id_3> alcohol<extra_id_4> A<extra_id_5> A. This<extra_id_6> I<extra_id_7><extra_id_52><extra_id_53></s>']\n```\n\n<Tip> \n\nRefer to [T5's documentation page](t5) for more tips, code examples and notebooks.\n</Tip>\n\n## UMT5Config\n\n[[autodoc]] UMT5Config\n\n## UMT5Model\n\n[[autodoc]] UMT5Model\n    - forward\n\n## UMT5ForConditionalGeneration\n\n[[autodoc]] UMT5ForConditionalGeneration\n    - forward\n\n## UMT5EncoderModel\n\n[[autodoc]] UMT5EncoderModel\n    - forward\n\n## UMT5ForSequenceClassification\n\n[[autodoc]] UMT5ForSequenceClassification\n    - forward\n\n## UMT5ForTokenClassification\n\n[[autodoc]] UMT5ForTokenClassification\n    - forward\n\n## UMT5ForQuestionAnswering\n\n[[autodoc]] UMT5ForQuestionAnswering\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7906494d-80df-439e-b971-6c4ef9d3e72e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/udop", "title": "UDOP", "tokens": 1659, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7906494d-80df-439e-b971-6c4ef9d3e72e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"57e8ee79-c268-4767-a48a-dd9c567b2c2c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"048971be410d1ad307cfdf125f112d6f67c3032221de1cbcc5e82cfd46a97328\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"35085f2a-82b9-4cd5-aa80-807341dd9e4b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2007babad6136620a3d34fc0dd0138a221b8b0743365908e84e9eccf6ef7441c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3279, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "doc_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "ref_doc_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "chroma:document": "# UDOP\n\n## Overview\n\nThe UDOP model was proposed in [Unifying Vision, Text, and Layout for Universal Document Processing](https://arxiv.org/abs/2212.02623) by Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal.\nUDOP adopts an encoder-decoder Transformer architecture based on [T5](t5) for document AI tasks like document image classification, document parsing and document visual question answering.\n\nThe abstract from the paper is the following:\n\nWe propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 9 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark (DUE).*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/udop_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> UDOP architecture. Taken from the <a href=\"https://arxiv.org/abs/2212.02623\">original paper.</a> </small>\n\n## Usage tips\n\n- In addition to *input_ids*, [`UdopForConditionalGeneration`] also expects the input `bbox`, which are\n  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such\n  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the\n  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000\n  scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:\n\n```python\nfrom PIL import Image\n\n# Document can be a png, jpg, etc. PDFs must be converted to images."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 35085f2a-82b9-4cd5-aa80-807341dd9e4b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/udop", "title": "UDOP", "tokens": 1659, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"35085f2a-82b9-4cd5-aa80-807341dd9e4b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"57e8ee79-c268-4767-a48a-dd9c567b2c2c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"048971be410d1ad307cfdf125f112d6f67c3032221de1cbcc5e82cfd46a97328\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7906494d-80df-439e-b971-6c4ef9d3e72e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b37c7afc4827e1d4ec11156ff2016a00df1dcec395c8409ea9e8ef32b669f0db\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"51f341f3-9c45-4c86-9b51-10a33d8ae1d0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3b3a1db6741f805ed01694f9c80a4cfdc5cdbc1fbb12fbda879ef6c652b827f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1917, \"end_char_idx\": 4870, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "doc_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "ref_doc_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2212.02623\">original paper.</a> </small>\n\n## Usage tips\n\n- In addition to *input_ids*, [`UdopForConditionalGeneration`] also expects the input `bbox`, which are\n  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such\n  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the\n  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000\n  scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:\n\n```python\nfrom PIL import Image\n\n# Document can be a png, jpg, etc. PDFs must be converted to images.\nimage = Image.open(name_of_your_document).convert(\"RGB\")\n\nwidth, height = image.size\n```\n\nOne can use [`UdopProcessor`] to prepare images and text for the model, which takes care of all of this. By default, this class uses the Tesseract engine to extract a list of words and boxes (coordinates) from a given document. Its functionality is equivalent to that of [`LayoutLMv3Processor`], hence it supports passing either `apply_ocr=False` in case you prefer to use your own OCR engine or `apply_ocr=True` in case you want the default OCR engine to be used. Refer to the [usage guide of LayoutLMv2](layoutlmv2#usage-layoutlmv2processor) regarding all possible use cases (the functionality of `UdopProcessor` is identical).\n\n- If using an own OCR engine of choice, one recommendation is Azure's [Read API](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/how-to/call-read-api), which supports so-called line segments. Use of segment position embeddings typically results in better performance.\n- At inference time, it's recommended to use the `generate` method to autoregressively generate text given a document image.\n- The model has been pre-trained on both self-supervised and supervised objectives. One can use the various task prefixes (prompts) used during pre-training to test out the out-of-the-box capabilities. For instance, the model can be prompted with \"Question answering. What is the date?\", as \"Question answering.\" is the task prefix used during pre-training for DocVQA. Refer to the [paper](https://arxiv.org/abs/2212.02623) (table 1) for all task prefixes."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 51f341f3-9c45-4c86-9b51-10a33d8ae1d0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/udop", "title": "UDOP", "tokens": 1659, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"51f341f3-9c45-4c86-9b51-10a33d8ae1d0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"57e8ee79-c268-4767-a48a-dd9c567b2c2c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"048971be410d1ad307cfdf125f112d6f67c3032221de1cbcc5e82cfd46a97328\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"35085f2a-82b9-4cd5-aa80-807341dd9e4b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9c7f0a43fe3957870772511a2df9a8e0020179c7cb9d1535423ba7e8ccf5d930\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b230b641-b711-4637-a0e7-c60209976081\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"05432d645f7a0cb96cde631a96879bcb4fd49942cb29691b93c96f6ce1968d96\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3246, \"end_char_idx\": 5826, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "doc_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "ref_doc_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "chroma:document": "PDFs must be converted to images.\nimage = Image.open(name_of_your_document).convert(\"RGB\")\n\nwidth, height = image.size\n```\n\nOne can use [`UdopProcessor`] to prepare images and text for the model, which takes care of all of this. By default, this class uses the Tesseract engine to extract a list of words and boxes (coordinates) from a given document. Its functionality is equivalent to that of [`LayoutLMv3Processor`], hence it supports passing either `apply_ocr=False` in case you prefer to use your own OCR engine or `apply_ocr=True` in case you want the default OCR engine to be used. Refer to the [usage guide of LayoutLMv2](layoutlmv2#usage-layoutlmv2processor) regarding all possible use cases (the functionality of `UdopProcessor` is identical).\n\n- If using an own OCR engine of choice, one recommendation is Azure's [Read API](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/how-to/call-read-api), which supports so-called line segments. Use of segment position embeddings typically results in better performance.\n- At inference time, it's recommended to use the `generate` method to autoregressively generate text given a document image.\n- The model has been pre-trained on both self-supervised and supervised objectives. One can use the various task prefixes (prompts) used during pre-training to test out the out-of-the-box capabilities. For instance, the model can be prompted with \"Question answering. What is the date?\", as \"Question answering.\" is the task prefix used during pre-training for DocVQA. Refer to the [paper](https://arxiv.org/abs/2212.02623) (table 1) for all task prefixes.\n- One can also fine-tune [`UdopEncoderModel`], which is the encoder-only part of UDOP, which can be seen as a LayoutLMv3-like Transformer encoder. For discriminative tasks, one can just add a linear classifier on top of it and fine-tune it on a labeled dataset.\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/UDOP).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with UDOP. If\nyou're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll\nreview it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Demo notebooks regarding UDOP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP) that show how\nto fine-tune UDOP on a custom dataset as well as inference."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b230b641-b711-4637-a0e7-c60209976081
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/udop", "title": "UDOP", "tokens": 1659, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b230b641-b711-4637-a0e7-c60209976081\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"57e8ee79-c268-4767-a48a-dd9c567b2c2c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"048971be410d1ad307cfdf125f112d6f67c3032221de1cbcc5e82cfd46a97328\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"51f341f3-9c45-4c86-9b51-10a33d8ae1d0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1510f92cd9bf9229593b36ce8d1475ffc02ad70560c6ad062623971670b79a7c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4213, \"end_char_idx\": 6445, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "doc_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "ref_doc_id": "57e8ee79-c268-4767-a48a-dd9c567b2c2c", "chroma:document": "Use of segment position embeddings typically results in better performance.\n- At inference time, it's recommended to use the `generate` method to autoregressively generate text given a document image.\n- The model has been pre-trained on both self-supervised and supervised objectives. One can use the various task prefixes (prompts) used during pre-training to test out the out-of-the-box capabilities. For instance, the model can be prompted with \"Question answering. What is the date?\", as \"Question answering.\" is the task prefix used during pre-training for DocVQA. Refer to the [paper](https://arxiv.org/abs/2212.02623) (table 1) for all task prefixes.\n- One can also fine-tune [`UdopEncoderModel`], which is the encoder-only part of UDOP, which can be seen as a LayoutLMv3-like Transformer encoder. For discriminative tasks, one can just add a linear classifier on top of it and fine-tune it on a labeled dataset.\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/UDOP).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with UDOP. If\nyou're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll\nreview it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Demo notebooks regarding UDOP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP) that show how\nto fine-tune UDOP on a custom dataset as well as inference. \ud83c\udf0e\n- [Document question answering task guide](../tasks/document_question_answering)\n\n## UdopConfig\n\n[[autodoc]] UdopConfig\n\n## UdopTokenizer\n\n[[autodoc]] UdopTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## UdopTokenizerFast\n\n[[autodoc]] UdopTokenizerFast\n\n## UdopProcessor\n\n[[autodoc]] UdopProcessor\n    - __call__\n\n## UdopModel\n\n[[autodoc]] UdopModel\n    - forward\n\n## UdopForConditionalGeneration\n\n[[autodoc]] UdopForConditionalGeneration\n    - forward\n\n## UdopEncoderModel\n\n[[autodoc]] UdopEncoderModel\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 61ebd7ff-a9b5-4a1f-97fb-565f20e3c6eb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama3", "title": "Llama3", "tokens": 1415, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"61ebd7ff-a9b5-4a1f-97fb-565f20e3c6eb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f4a76e2b-85f3-4512-bc29-71db21efd31f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2c36a307f6db561d9ce6fffdacc8d81f809d3a1714d09c6e3cac163db98dc1b1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d5890ea5-e0d1-4857-b821-dcef28f70546\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ac8c6fd5b8eba52c1f6759982d23fc8f52cdb7ad9c736d321f30c0f55a4fc6b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3199, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f4a76e2b-85f3-4512-bc29-71db21efd31f", "doc_id": "f4a76e2b-85f3-4512-bc29-71db21efd31f", "ref_doc_id": "f4a76e2b-85f3-4512-bc29-71db21efd31f", "chroma:document": "# Llama3\n\n```py3\nimport transformers\nimport torch\n\nmodel_id = \"meta-llama/Meta-Llama-3-8B\"\n\npipeline = transformers.pipeline(\"text-generation\", model=model_id, model_kwargs={\"torch_dtype\": torch.bfloat16}, device_map=\"auto\")\npipeline(\"Hey how are you doing today?\")\n```\n\n## Overview\n\nThe Llama3 model was proposed in [Introducing Meta Llama 3: The most capable openly available LLM to date](https://ai.meta.com/blog/meta-llama-3/) by the meta AI team.\n\nThe abstract from the blogpost is the following:\n\n*Today, we\u2019re excited to share the first two models of the next generation of Llama, Meta Llama 3, available for broad use. This release features pretrained and instruction-fine-tuned language models with 8B and 70B parameters that can support a broad range of use cases. This next generation of Llama demonstrates state-of-the-art performance on a wide range of industry benchmarks and offers new capabilities, including improved reasoning. We believe these are the best open source models of their class, period. In support of our longstanding open approach, we\u2019re putting Llama 3 in the hands of the community. We want to kickstart the next wave of innovation in AI across the stack\u2014from applications to developer tools to evals to inference optimizations and more. We can\u2019t wait to see what you build and look forward to your feedback.*\n\nCheckout all Llama3 model checkpoints [here](https://huggingface.co/models?search=llama3).\nThe original code of the authors can be found [here](https://github.com/meta-llama/llama3).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe `Llama3` models were trained using `bfloat16`, but the original inference uses `float16`. The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\n\nTips:\n\n- Weights for the Llama3 models can be obtained by filling out [this form](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)\n- The architecture is exactly the same as Llama2.\n- The tokenizer is a BPE model based on [tiktoken](https://github.com/openai/tiktoken) (vs the one based on sentencepiece implementation for Llama2). The main difference that it ignores BPE merge rules when an input token is part of the vocab. This means that if no merge exist to produce `\"hugging\"`, instead of having the smallest units, like `[\"hug\",\"ging\"] form 2 tokens, if `\"hugging\"` is part of the vocab, it will be automatically returned as a token.\n- The original model uses `pad_id = -1` which means that there is no padding token."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d5890ea5-e0d1-4857-b821-dcef28f70546
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama3", "title": "Llama3", "tokens": 1415, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d5890ea5-e0d1-4857-b821-dcef28f70546\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f4a76e2b-85f3-4512-bc29-71db21efd31f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2c36a307f6db561d9ce6fffdacc8d81f809d3a1714d09c6e3cac163db98dc1b1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"61ebd7ff-a9b5-4a1f-97fb-565f20e3c6eb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3b88e1c5eb23cedba9faacf13dfedd943ebfd4341a8fdb1f44d4c6ef94cd21ea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"10703d16-d036-4d75-9e80-f018b820b807\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"78edebbbb2fda98dbdf33156ba4d8c3f88527a5ce2c2425bb0d0ca556e264b38\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1660, \"end_char_idx\": 4885, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f4a76e2b-85f3-4512-bc29-71db21efd31f", "doc_id": "f4a76e2b-85f3-4512-bc29-71db21efd31f", "ref_doc_id": "f4a76e2b-85f3-4512-bc29-71db21efd31f", "chroma:document": "The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\n\nTips:\n\n- Weights for the Llama3 models can be obtained by filling out [this form](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)\n- The architecture is exactly the same as Llama2.\n- The tokenizer is a BPE model based on [tiktoken](https://github.com/openai/tiktoken) (vs the one based on sentencepiece implementation for Llama2). The main difference that it ignores BPE merge rules when an input token is part of the vocab. This means that if no merge exist to produce `\"hugging\"`, instead of having the smallest units, like `[\"hug\",\"ging\"] form 2 tokens, if `\"hugging\"` is part of the vocab, it will be automatically returned as a token.\n- The original model uses `pad_id = -1` which means that there is no padding token. We can't have the same logic, make sure to add a padding token using `tokenizer.add_special_tokens({\"pad_token\":\"<pad>\"})` and resize the token embedding accordingly. You should also set the `model.config.pad_token_id`. The `embed_tokens` layer of the model is initialized with `self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.config.padding_idx)`, which makes sure that encoding the padding token will output zeros, so passing it when initializing is recommended.\n- The original checkpoint can be converted using the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path --llama_version 3\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"/output/path\")\nmodel = AutoModelForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 75B model, it's thus 145GB of RAM needed.\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 10703d16-d036-4d75-9e80-f018b820b807
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama3", "title": "Llama3", "tokens": 1415, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"10703d16-d036-4d75-9e80-f018b820b807\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f4a76e2b-85f3-4512-bc29-71db21efd31f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2c36a307f6db561d9ce6fffdacc8d81f809d3a1714d09c6e3cac163db98dc1b1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d5890ea5-e0d1-4857-b821-dcef28f70546\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1c108ce2ac8c1d8ea4b09d44b53e83c361280d03bd4d77e043d54b6eeba13e0c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3200, \"end_char_idx\": 5285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f4a76e2b-85f3-4512-bc29-71db21efd31f", "doc_id": "f4a76e2b-85f3-4512-bc29-71db21efd31f", "ref_doc_id": "f4a76e2b-85f3-4512-bc29-71db21efd31f", "chroma:document": "We can't have the same logic, make sure to add a padding token using `tokenizer.add_special_tokens({\"pad_token\":\"<pad>\"})` and resize the token embedding accordingly. You should also set the `model.config.pad_token_id`. The `embed_tokens` layer of the model is initialized with `self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.config.padding_idx)`, which makes sure that encoding the padding token will output zeros, so passing it when initializing is recommended.\n- The original checkpoint can be converted using the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path --llama_version 3\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"/output/path\")\nmodel = AutoModelForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 75B model, it's thus 145GB of RAM needed.\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.\n\n## Resources\nA ton of cool resources are already available on the documentation page of [Llama2](./llama2), inviting contributors to add new resources curated for Llama3 here! \ud83e\udd17"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 65ebf929-f2f9-4efe-aeb2-7707bb896387
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert-japanese", "title": "BertJapanese", "tokens": 670, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"65ebf929-f2f9-4efe-aeb2-7707bb896387\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-japanese\", \"title\": \"BertJapanese\", \"tokens\": 670, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"356547a6-f05e-40b4-9d28-2ef3a7a7f1ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-japanese\", \"title\": \"BertJapanese\", \"tokens\": 670, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"34c3999e9470768dfc788f23fc66e1952bb83129a71c4160ac3ea9618b882398\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1862, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "356547a6-f05e-40b4-9d28-2ef3a7a7f1ad", "doc_id": "356547a6-f05e-40b4-9d28-2ef3a7a7f1ad", "ref_doc_id": "356547a6-f05e-40b4-9d28-2ef3a7a7f1ad", "chroma:document": "# BertJapanese\n\n## Overview\n\nThe BERT models trained on Japanese text.\n\nThere are models with two different tokenization methods:\n\n- Tokenize with MeCab and WordPiece. This requires some extra dependencies, [fugashi](https://github.com/polm/fugashi) which is a wrapper around [MeCab](https://taku910.github.io/mecab/).\n- Tokenize into characters.\n\nTo use *MecabTokenizer*, you should `pip install transformers[\"ja\"]` (or `pip install -e .[\"ja\"]` if you install\nfrom source) to install dependencies.\n\nSee [details on cl-tohoku repository](https://github.com/cl-tohoku/bert-japanese).\n\nExample of using a model with MeCab and WordPiece tokenization:\n\n```python\n>>> import torch\n>>> from transformers import AutoModel, AutoTokenizer\n\n>>> bertjapanese = AutoModel.from_pretrained(\"cl-tohoku/bert-base-japanese\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"cl-tohoku/bert-base-japanese\")\n\n>>> ## Input Japanese Text\n>>> line = \"\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\u3002\"\n\n>>> inputs = tokenizer(line, return_tensors=\"pt\")\n\n>>> print(tokenizer.decode(inputs[\"input_ids\"][0]))\n[CLS] \u543e\u8f29 \u306f \u732b \u3067 \u3042\u308b \u3002 [SEP]\n\n>>> outputs = bertjapanese(**inputs)\n```\n\nExample of using a model with Character tokenization:\n\n```python\n>>> bertjapanese = AutoModel.from_pretrained(\"cl-tohoku/bert-base-japanese-char\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"cl-tohoku/bert-base-japanese-char\")\n\n>>> ## Input Japanese Text\n>>> line = \"\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\u3002\"\n\n>>> inputs = tokenizer(line, return_tensors=\"pt\")\n\n>>> print(tokenizer.decode(inputs[\"input_ids\"][0]))\n[CLS] \u543e \u8f29 \u306f \u732b \u3067 \u3042 \u308b \u3002 [SEP]\n\n>>> outputs = bertjapanese(**inputs)\n```\n\nThis model was contributed by [cl-tohoku](https://huggingface.co/cl-tohoku).\n\n<Tip> \n\nThis implementation is the same as BERT, except for tokenization method. Refer to [BERT documentation](bert) for \nAPI reference information.  \n\n</Tip>\n\n\n## BertJapaneseTokenizer\n\n[[autodoc]] BertJapaneseTokenizer"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 406e1693-98cd-471b-a758-aa99b3b7ffcb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/align", "title": "ALIGN", "tokens": 1281, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"406e1693-98cd-471b-a758-aa99b3b7ffcb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"849adcf2-1f40-44f2-a9fb-a713e6bcb5c4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d73cb1df62e0391c744e0543d87805f667e500553fb06d91e9058ad45e06ff84\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4572c405-10fe-41d6-82a2-364dc0eca2e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"23039f70f72a47ad99b8b4fc0457338739dc4ea1a0e4b3abdad3e265f4a7e302\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3296, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "849adcf2-1f40-44f2-a9fb-a713e6bcb5c4", "doc_id": "849adcf2-1f40-44f2-a9fb-a713e6bcb5c4", "ref_doc_id": "849adcf2-1f40-44f2-a9fb-a713e6bcb5c4", "chroma:document": "# ALIGN\n\n## Overview\n\nThe ALIGN model was proposed in [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://arxiv.org/abs/2102.05918) by Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, Tom Duerig. ALIGN is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image classification. ALIGN features a dual-encoder architecture with [EfficientNet](efficientnet) as its vision encoder and [BERT](bert) as its text encoder, and learns to align visual and text representations with contrastive learning. Unlike previous work, ALIGN leverages a massive noisy dataset and shows that the scale of the corpus can be used to achieve SOTA representations with a simple recipe.\n\nThe abstract from the paper is the following:\n\n*Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.*\n\nThis model was contributed by [Alara Dirik](https://huggingface.co/adirik).\nThe original code is not released, this implementation is based on the Kakao Brain implementation based on the original paper.\n\n## Usage example\n\nALIGN uses EfficientNet to get visual features and BERT to get the text features. Both the text and visual features are then projected to a latent space with identical dimension. The dot product between the projected image and text features is then used as a similarity score.\n\n[`AlignProcessor`] wraps [`EfficientNetImageProcessor`] and [`BertTokenizer`] into a single instance to both encode the text and preprocess the images. The following example shows how to get the image-text similarity scores using [`AlignProcessor`] and [`AlignModel`]."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4572c405-10fe-41d6-82a2-364dc0eca2e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/align", "title": "ALIGN", "tokens": 1281, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4572c405-10fe-41d6-82a2-364dc0eca2e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"849adcf2-1f40-44f2-a9fb-a713e6bcb5c4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d73cb1df62e0391c744e0543d87805f667e500553fb06d91e9058ad45e06ff84\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"406e1693-98cd-471b-a758-aa99b3b7ffcb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a205e192e8a59273b8ea4173c59d2df0834e8678fb79e598cd41f4d61b36f69c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b303064c-bfd0-405a-bdc0-2a440616a5d6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"971c9c808d8a25ede08518cb71ba0b6b5cc1519e88f514c92642ac688c0d1153\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1327, \"end_char_idx\": 4712, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "849adcf2-1f40-44f2-a9fb-a713e6bcb5c4", "doc_id": "849adcf2-1f40-44f2-a9fb-a713e6bcb5c4", "ref_doc_id": "849adcf2-1f40-44f2-a9fb-a713e6bcb5c4", "chroma:document": "For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.*\n\nThis model was contributed by [Alara Dirik](https://huggingface.co/adirik).\nThe original code is not released, this implementation is based on the Kakao Brain implementation based on the original paper.\n\n## Usage example\n\nALIGN uses EfficientNet to get visual features and BERT to get the text features. Both the text and visual features are then projected to a latent space with identical dimension. The dot product between the projected image and text features is then used as a similarity score.\n\n[`AlignProcessor`] wraps [`EfficientNetImageProcessor`] and [`BertTokenizer`] into a single instance to both encode the text and preprocess the images. The following example shows how to get the image-text similarity scores using [`AlignProcessor`] and [`AlignModel`].\n\n```python\nimport requests\nimport torch\nfrom PIL import Image\nfrom transformers import AlignProcessor, AlignModel\n\nprocessor = AlignProcessor.from_pretrained(\"kakaobrain/align-base\")\nmodel = AlignModel.from_pretrained(\"kakaobrain/align-base\")\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\ncandidate_labels = [\"an image of a cat\", \"an image of a dog\"]\n\ninputs = processor(text=candidate_labels, images=image, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\n# this is the image-text similarity score\nlogits_per_image = outputs.logits_per_image\n\n# we can take the softmax to get the label probabilities\nprobs = logits_per_image.softmax(dim=1)\nprint(probs)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ALIGN.\n\n- A blog post on [ALIGN and the COYO-700M dataset](https://huggingface.co/blog/vit-align).\n- A zero-shot image classification [demo](https://huggingface.co/spaces/adirik/ALIGN-zero-shot-image-classification).\n- [Model card](https://huggingface.co/kakaobrain/align-base) of `kakaobrain/align-base` model.\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it. The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b303064c-bfd0-405a-bdc0-2a440616a5d6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/align", "title": "ALIGN", "tokens": 1281, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b303064c-bfd0-405a-bdc0-2a440616a5d6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"849adcf2-1f40-44f2-a9fb-a713e6bcb5c4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d73cb1df62e0391c744e0543d87805f667e500553fb06d91e9058ad45e06ff84\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4572c405-10fe-41d6-82a2-364dc0eca2e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5dfceeea1a7b227c57516cfa82fff2a442ebc632573017c21f44cd62af6024f3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3028, \"end_char_idx\": 5161, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "849adcf2-1f40-44f2-a9fb-a713e6bcb5c4", "doc_id": "849adcf2-1f40-44f2-a9fb-a713e6bcb5c4", "ref_doc_id": "849adcf2-1f40-44f2-a9fb-a713e6bcb5c4", "chroma:document": "[`AlignProcessor`] wraps [`EfficientNetImageProcessor`] and [`BertTokenizer`] into a single instance to both encode the text and preprocess the images. The following example shows how to get the image-text similarity scores using [`AlignProcessor`] and [`AlignModel`].\n\n```python\nimport requests\nimport torch\nfrom PIL import Image\nfrom transformers import AlignProcessor, AlignModel\n\nprocessor = AlignProcessor.from_pretrained(\"kakaobrain/align-base\")\nmodel = AlignModel.from_pretrained(\"kakaobrain/align-base\")\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\ncandidate_labels = [\"an image of a cat\", \"an image of a dog\"]\n\ninputs = processor(text=candidate_labels, images=image, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\n# this is the image-text similarity score\nlogits_per_image = outputs.logits_per_image\n\n# we can take the softmax to get the label probabilities\nprobs = logits_per_image.softmax(dim=1)\nprint(probs)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ALIGN.\n\n- A blog post on [ALIGN and the COYO-700M dataset](https://huggingface.co/blog/vit-align).\n- A zero-shot image classification [demo](https://huggingface.co/spaces/adirik/ALIGN-zero-shot-image-classification).\n- [Model card](https://huggingface.co/kakaobrain/align-base) of `kakaobrain/align-base` model.\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it. The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## AlignConfig\n\n[[autodoc]] AlignConfig\n    - from_text_vision_configs\n\n## AlignTextConfig\n\n[[autodoc]] AlignTextConfig\n\n## AlignVisionConfig\n\n[[autodoc]] AlignVisionConfig\n\n## AlignProcessor\n\n[[autodoc]] AlignProcessor\n\n## AlignModel\n\n[[autodoc]] AlignModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## AlignTextModel\n\n[[autodoc]] AlignTextModel\n    - forward\n\n## AlignVisionModel\n\n[[autodoc]] AlignVisionModel\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: bd8f855c-ceee-4223-b0b3-05821ece62e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dialogpt", "title": "DialoGPT", "tokens": 729, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bd8f855c-ceee-4223-b0b3-05821ece62e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dialogpt\", \"title\": \"DialoGPT\", \"tokens\": 729, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"278d11ba-010b-4400-8188-a1590b630b62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dialogpt\", \"title\": \"DialoGPT\", \"tokens\": 729, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"acd00c53e1f6f865d113365069be34dd15e5153ec00030034f686aaade1a8b58\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2398, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "278d11ba-010b-4400-8188-a1590b630b62", "doc_id": "278d11ba-010b-4400-8188-a1590b630b62", "ref_doc_id": "278d11ba-010b-4400-8188-a1590b630b62", "chroma:document": "# DialoGPT\n\n## Overview\n\nDialoGPT was proposed in [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://arxiv.org/abs/1911.00536) by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao,\nJianfeng Gao, Jingjing Liu, Bill Dolan. It's a GPT2 Model trained on 147M conversation-like exchanges extracted from\nReddit.\n\nThe abstract from the paper is the following:\n\n*We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained\ntransformer). Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning\nfrom 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human\nboth in terms of automatic and human evaluation in single-turn dialogue settings. We show that conversational systems\nthat leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline\nsystems. The pre-trained model and training pipeline are publicly released to facilitate research into neural response\ngeneration and the development of more intelligent open-domain dialogue systems.*\n\nThe original code can be found [here](https://github.com/microsoft/DialoGPT).\n\n## Usage tips\n\n- DialoGPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather\n  than the left.\n- DialoGPT was trained with a causal language modeling (CLM) objective on conversational data and is therefore powerful\n  at response generation in open-domain dialogue systems.\n- DialoGPT enables the user to create a chat bot in just 10 lines of code as shown on [DialoGPT's model card](https://huggingface.co/microsoft/DialoGPT-medium).\n\nTraining:\n\nIn order to train or fine-tune DialoGPT, one can use causal language modeling training. To cite the official paper: *We\nfollow the OpenAI GPT-2 to model a multiturn dialogue session as a long text and frame the generation task as language\nmodeling. We first concatenate all dialog turns within a dialogue session into a long text x_1,..., x_N (N is the\nsequence length), ended by the end-of-text token.* For more information please confer to the original paper.\n\n<Tip>\n\nDialoGPT's architecture is based on the GPT2 model, refer to [GPT2's documentation page](gpt2) for API reference and examples.\n\n</Tip>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 879859dc-eced-459c-a3a7-126e76a5d743
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/regnet", "title": "RegNet", "tokens": 981, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"879859dc-eced-459c-a3a7-126e76a5d743\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"757f6ec8-d9c6-4e1d-bfa6-406c885f56b8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"25666284c633cc766faa610f9b3e9860b36485007df2712da6302236fa8fa7cc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5aacd41a-9067-4a89-a740-20ed8bae0b42\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4f0ee605765d5c3213adb84b51d56f0286fbd26485f906658e943014a859d03d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3010, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "757f6ec8-d9c6-4e1d-bfa6-406c885f56b8", "doc_id": "757f6ec8-d9c6-4e1d-bfa6-406c885f56b8", "ref_doc_id": "757f6ec8-d9c6-4e1d-bfa6-406c885f56b8", "chroma:document": "# RegNet\n\n## Overview\n\nThe RegNet model was proposed in [Designing Network Design Spaces](https://arxiv.org/abs/2003.13678) by Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Doll\u00e1r.\n\nThe authors design search spaces to perform Neural Architecture Search (NAS). They first start from a high dimensional search space and iteratively reduce the search space by empirically applying constraints based on the best-performing models sampled by the current search space.\n\nThe abstract from the paper is the following:\n\n*In this work, we present a new network design paradigm. Our goal is to help advance the understanding of network design and discover design principles that generalize across settings. Instead of focusing on designing individual network instances, we design network design spaces that parametrize populations of networks. The overall process is analogous to classic manual design of networks, but elevated to the design space level. Using our methodology we explore the structure aspect of network design and arrive at a low-dimensional design space consisting of simple, regular networks that we call RegNet. The core insight of the RegNet parametrization is surprisingly simple: widths and depths of good networks can be explained by a quantized linear function. We analyze the RegNet design space and arrive at interesting findings that do not match the current practice of network design. The RegNet design space provides simple and fast networks that work well across a wide range of flop regimes. Under comparable training settings and flops, the RegNet models outperform the popular EfficientNet models while being up to 5x faster on GPUs.*\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of the model\nwas contributed by [sayakpaul](https://huggingface.co/sayakpaul) and [ariG23498](https://huggingface.co/ariG23498).\nThe original code can be found [here](https://github.com/facebookresearch/pycls).\n\nThe huge 10B model from [Self-supervised Pretraining of Visual Features in the Wild](https://arxiv.org/abs/2103.01988), \ntrained on  one billion Instagram images, is available on the [hub](https://huggingface.co/facebook/regnet-y-10b-seer)\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RegNet.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`RegNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5aacd41a-9067-4a89-a740-20ed8bae0b42
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/regnet", "title": "RegNet", "tokens": 981, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5aacd41a-9067-4a89-a740-20ed8bae0b42\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"757f6ec8-d9c6-4e1d-bfa6-406c885f56b8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"25666284c633cc766faa610f9b3e9860b36485007df2712da6302236fa8fa7cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"879859dc-eced-459c-a3a7-126e76a5d743\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61ea6b66784f3831368d140e72601bfa449404b6893b1ab83254ac47e767eac5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1304, \"end_char_idx\": 3582, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "757f6ec8-d9c6-4e1d-bfa6-406c885f56b8", "doc_id": "757f6ec8-d9c6-4e1d-bfa6-406c885f56b8", "ref_doc_id": "757f6ec8-d9c6-4e1d-bfa6-406c885f56b8", "chroma:document": "We analyze the RegNet design space and arrive at interesting findings that do not match the current practice of network design. The RegNet design space provides simple and fast networks that work well across a wide range of flop regimes. Under comparable training settings and flops, the RegNet models outperform the popular EfficientNet models while being up to 5x faster on GPUs.*\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of the model\nwas contributed by [sayakpaul](https://huggingface.co/sayakpaul) and [ariG23498](https://huggingface.co/ariG23498).\nThe original code can be found [here](https://github.com/facebookresearch/pycls).\n\nThe huge 10B model from [Self-supervised Pretraining of Visual Features in the Wild](https://arxiv.org/abs/2103.01988), \ntrained on  one billion Instagram images, is available on the [hub](https://huggingface.co/facebook/regnet-y-10b-seer)\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RegNet.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`RegNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## RegNetConfig\n\n[[autodoc]] RegNetConfig\n\n<frameworkcontent>\n<pt>\n\n## RegNetModel\n\n[[autodoc]] RegNetModel\n    - forward\n\n## RegNetForImageClassification\n\n[[autodoc]] RegNetForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFRegNetModel\n\n[[autodoc]] TFRegNetModel\n    - call\n\n## TFRegNetForImageClassification\n\n[[autodoc]] TFRegNetForImageClassification\n    - call\n\n</tf>\n<jax>\n\n## FlaxRegNetModel\n\n[[autodoc]] FlaxRegNetModel\n    - __call__\n\n## FlaxRegNetForImageClassification\n\n[[autodoc]] FlaxRegNetForImageClassification\n    - __call__\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4c87978a-ecd5-4604-a1b1-80ea881b54e6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything", "title": "Depth Anything", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4c87978a-ecd5-4604-a1b1-80ea881b54e6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ca2de465-e571-468d-9311-c5e4ed006f39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eb6c1de1530b1ad4af5fcd404d4b89d0e7407f1db35360580ccaafdb82361b8b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fa862185-a70a-498a-b634-40a146429e74\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fbd9d993ddbaa1750747c29fc7e5f658fcd5d5f385e50b6d419115539345120b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2703, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ca2de465-e571-468d-9311-c5e4ed006f39", "doc_id": "ca2de465-e571-468d-9311-c5e4ed006f39", "ref_doc_id": "ca2de465-e571-468d-9311-c5e4ed006f39", "chroma:document": "# Depth Anything\n\n## Overview\n\nThe Depth Anything model was proposed in [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://arxiv.org/abs/2401.10891) by Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao. Depth Anything is based on the [DPT](dpt) architecture, trained on ~62 million images, obtaining state-of-the-art results for both relative and absolute depth estimation.\n\n<Tip>\n\n[Depth Anything V2](depth_anything_v2) was released in June 2024. It uses the same architecture as Depth Anything and therefore it is compatible with all code examples and existing workflows. However, it leverages synthetic data and a larger capacity teacher model to achieve much finer and robust depth predictions.\n\n</Tip>\n\nThe abstract from the paper is the following:\n\n*This work presents Depth Anything, a highly practical solution for robust monocular depth estimation. Without pursuing novel technical modules, we aim to build a simple yet powerful foundation model dealing with any images under any circumstances. To this end, we scale up the dataset by designing a data engine to collect and automatically annotate large-scale unlabeled data (~62M), which significantly enlarges the data coverage and thus is able to reduce the generalization error. We investigate two simple yet effective strategies that make data scaling-up promising. First, a more challenging optimization target is created by leveraging data augmentation tools. It compels the model to actively seek extra visual knowledge and acquire robust representations. Second, an auxiliary supervision is developed to enforce the model to inherit rich semantic priors from pre-trained encoders. We evaluate its zero-shot capabilities extensively, including six public datasets and randomly captured photos. It demonstrates impressive generalization ability. Further, through fine-tuning it with metric depth information from NYUv2 and KITTI, new SOTAs are set. Our better depth model also results in a better depth-conditioned ControlNet.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Depth Anything overview. Taken from the <a href=\"https://arxiv.org/abs/2401.10891\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/LiheYoung/Depth-Anything).\n\n## Usage example\n\nThere are 2 main ways to use Depth Anything: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: fa862185-a70a-498a-b634-40a146429e74
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything", "title": "Depth Anything", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fa862185-a70a-498a-b634-40a146429e74\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ca2de465-e571-468d-9311-c5e4ed006f39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eb6c1de1530b1ad4af5fcd404d4b89d0e7407f1db35360580ccaafdb82361b8b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4c87978a-ecd5-4604-a1b1-80ea881b54e6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bb6bc6cbf27979716ade83b81609cb279f51a782357c8786c0fc8d54d664fa7c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5e21549c-a206-44cb-bc04-a3a3cf039217\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3ee77e0b0fead26cdefe4e9d335f68f62d6c752467ac8ae293802127f3af7c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 914, \"end_char_idx\": 4524, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ca2de465-e571-468d-9311-c5e4ed006f39", "doc_id": "ca2de465-e571-468d-9311-c5e4ed006f39", "ref_doc_id": "ca2de465-e571-468d-9311-c5e4ed006f39", "chroma:document": "Without pursuing novel technical modules, we aim to build a simple yet powerful foundation model dealing with any images under any circumstances. To this end, we scale up the dataset by designing a data engine to collect and automatically annotate large-scale unlabeled data (~62M), which significantly enlarges the data coverage and thus is able to reduce the generalization error. We investigate two simple yet effective strategies that make data scaling-up promising. First, a more challenging optimization target is created by leveraging data augmentation tools. It compels the model to actively seek extra visual knowledge and acquire robust representations. Second, an auxiliary supervision is developed to enforce the model to inherit rich semantic priors from pre-trained encoders. We evaluate its zero-shot capabilities extensively, including six public datasets and randomly captured photos. It demonstrates impressive generalization ability. Further, through fine-tuning it with metric depth information from NYUv2 and KITTI, new SOTAs are set. Our better depth model also results in a better depth-conditioned ControlNet.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Depth Anything overview. Taken from the <a href=\"https://arxiv.org/abs/2401.10891\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/LiheYoung/Depth-Anything).\n\n## Usage example\n\nThere are 2 main ways to use Depth Anything: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself.\n\n### Pipeline API\n\nThe pipeline allows to use the model in a few lines of code:\n\n```python\n>>> from transformers import pipeline\n>>> from PIL import Image\n>>> import requests\n\n>>> # load pipe\n>>> pipe = pipeline(task=\"depth-estimation\", model=\"LiheYoung/depth-anything-small-hf\")\n\n>>> # load image\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> # inference\n>>> depth = pipe(image)[\"depth\"]\n```\n\n### Using the model yourself\n\nIf you want to do the pre- and postprocessing yourself, here's how to do that:\n\n```python\n>>> from transformers import AutoImageProcessor, AutoModelForDepthEstimation\n>>> import torch\n>>> import numpy as np\n>>> from PIL import Image\n>>> import requests\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\"LiheYoung/depth-anything-small-hf\")\n>>> model = AutoModelForDepthEstimation.from_pretrained(\"LiheYoung/depth-anything-small-hf\")\n\n>>> # prepare image for the model\n>>> inputs = image_processor(images=image, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n...     predicted_depth = outputs.predicted_depth\n\n>>> # interpolate to original size\n>>> prediction = torch.nn.functional.interpolate(\n...     predicted_depth.unsqueeze(1),\n...     size=image.size[::-1],\n...     mode=\"bicubic\",\n...     align_corners=False,\n... )\n\n>>> # visualize the prediction\n>>> output = prediction.squeeze().cpu().numpy()\n>>> formatted = (output * 255 / np.max(output)).astype(\"uint8\")\n>>> depth = Image.fromarray(formatted)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Depth Anything."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5e21549c-a206-44cb-bc04-a3a3cf039217
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything", "title": "Depth Anything", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5e21549c-a206-44cb-bc04-a3a3cf039217\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ca2de465-e571-468d-9311-c5e4ed006f39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eb6c1de1530b1ad4af5fcd404d4b89d0e7407f1db35360580ccaafdb82361b8b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fa862185-a70a-498a-b634-40a146429e74\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ea96821cc2d41b2a57ac137d3326e9b3fbdb9df477f5d737bb9ea437f1f5050d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4526, \"end_char_idx\": 5210, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ca2de465-e571-468d-9311-c5e4ed006f39", "doc_id": "ca2de465-e571-468d-9311-c5e4ed006f39", "ref_doc_id": "ca2de465-e571-468d-9311-c5e4ed006f39", "chroma:document": "- [Monocular depth estimation task guide](../tasks/depth_estimation)\n- A notebook showcasing inference with [`DepthAnythingForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Depth%20Anything/Predicting_depth_in_an_image_with_Depth_Anything.ipynb). \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DepthAnythingConfig\n\n[[autodoc]] DepthAnythingConfig\n\n## DepthAnythingForDepthEstimation\n\n[[autodoc]] DepthAnythingForDepthEstimation\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 49f70384-ed85-4c5a-a50e-eebf4934bd88
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yolos", "title": "YOLOS", "tokens": 1419, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"49f70384-ed85-4c5a-a50e-eebf4934bd88\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"53ffe6c7-1d35-4be8-abf1-58dd5b7c9468\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15e603461d5b610e0da53196578808c3a31bc7cd2f3d46b2d21862f97944ca20\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0237995c-9430-4994-84a2-2b052c713c7e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c39c72ec2ac702891465730cf21f151721bb963b1b983f7f97b36876279dcd5b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3068, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "53ffe6c7-1d35-4be8-abf1-58dd5b7c9468", "doc_id": "53ffe6c7-1d35-4be8-abf1-58dd5b7c9468", "ref_doc_id": "53ffe6c7-1d35-4be8-abf1-58dd5b7c9468", "chroma:document": "# YOLOS\n\n## Overview\n\nThe YOLOS model was proposed in [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://arxiv.org/abs/2106.00666) by Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, Wenyu Liu.\nYOLOS proposes to just leverage the plain [Vision Transformer (ViT)](vit) for object detection, inspired by DETR. It turns out that a base-sized encoder-only Transformer can also achieve 42 AP on COCO, similar to DETR and much more complex frameworks such as Faster R-CNN.\n\nThe abstract from the paper is the following:\n\n*Can Transformer perform 2D object- and region-level recognition from a pure sequence-to-sequence perspective with minimal knowledge about the 2D spatial structure? To answer this question, we present You Only Look at One Sequence (YOLOS), a series of object detection models based on the vanilla Vision Transformer with the fewest possible modifications, region priors, as well as inductive biases of the target task. We find that YOLOS pre-trained on the mid-sized ImageNet-1k dataset only can already achieve quite competitive performance on the challenging COCO object detection benchmark, e.g., YOLOS-Base directly adopted from BERT-Base architecture can obtain 42.0 box AP on COCO val. We also discuss the impacts as well as limitations of current pre-train schemes and model scaling strategies for Transformer in vision through YOLOS.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/yolos_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> YOLOS architecture. Taken from the <a href=\"https://arxiv.org/abs/2106.00666\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/hustvl/YOLOS).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import AutoModelForObjectDetection\nmodel = AutoModelForObjectDetection.from_pretrained(\"hustvl/yolos-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `hustvl/yolos-base` model, we saw the following speedups during inference."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0237995c-9430-4994-84a2-2b052c713c7e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yolos", "title": "YOLOS", "tokens": 1419, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0237995c-9430-4994-84a2-2b052c713c7e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"53ffe6c7-1d35-4be8-abf1-58dd5b7c9468\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15e603461d5b610e0da53196578808c3a31bc7cd2f3d46b2d21862f97944ca20\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"49f70384-ed85-4c5a-a50e-eebf4934bd88\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6ccd7b73081886a71771ece4cd30c8dd5c43fed6766da0e9b06fc43122d0fafa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9ad03816-73fa-44b1-acc8-d370d4afff25\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b06da1c0fb6ee413ad116d401252b45a2600973d2696bb43f10033d73bc1a34a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1614, \"end_char_idx\": 4956, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "53ffe6c7-1d35-4be8-abf1-58dd5b7c9468", "doc_id": "53ffe6c7-1d35-4be8-abf1-58dd5b7c9468", "ref_doc_id": "53ffe6c7-1d35-4be8-abf1-58dd5b7c9468", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2106.00666\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/hustvl/YOLOS).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import AutoModelForObjectDetection\nmodel = AutoModelForObjectDetection.from_pretrained(\"hustvl/yolos-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `hustvl/yolos-base` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                       106 |                                        76 |                      1.39 |\n|            2 |                                       154 |                                        90 |                      1.71 |\n|            4 |                                       222 |                                       116 |                      1.91 |\n|            8 |                                       368 |                                       168 |                      2.19 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with YOLOS.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- All example notebooks illustrating inference + fine-tuning [`YolosForObjectDetection`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/YOLOS).\n- Scripts for finetuning [`YolosForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<Tip>\n\nUse [`YolosImageProcessor`] for preparing images (and optional targets) for the model. Contrary to [DETR](detr), YOLOS doesn't require a `pixel_mask` to be created."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9ad03816-73fa-44b1-acc8-d370d4afff25
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yolos", "title": "YOLOS", "tokens": 1419, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9ad03816-73fa-44b1-acc8-d370d4afff25\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"53ffe6c7-1d35-4be8-abf1-58dd5b7c9468\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15e603461d5b610e0da53196578808c3a31bc7cd2f3d46b2d21862f97944ca20\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0237995c-9430-4994-84a2-2b052c713c7e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8cf7191a8d9c82f10f4bab69f76f567b509eaee7e77b88a43c87a7c58b5f1d97\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3070, \"end_char_idx\": 5379, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "53ffe6c7-1d35-4be8-abf1-58dd5b7c9468", "doc_id": "53ffe6c7-1d35-4be8-abf1-58dd5b7c9468", "ref_doc_id": "53ffe6c7-1d35-4be8-abf1-58dd5b7c9468", "chroma:document": "|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                       106 |                                        76 |                      1.39 |\n|            2 |                                       154 |                                        90 |                      1.71 |\n|            4 |                                       222 |                                       116 |                      1.91 |\n|            8 |                                       368 |                                       168 |                      2.19 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with YOLOS.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- All example notebooks illustrating inference + fine-tuning [`YolosForObjectDetection`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/YOLOS).\n- Scripts for finetuning [`YolosForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<Tip>\n\nUse [`YolosImageProcessor`] for preparing images (and optional targets) for the model. Contrary to [DETR](detr), YOLOS doesn't require a `pixel_mask` to be created.\n\n</Tip>\n\n## YolosConfig\n\n[[autodoc]] YolosConfig\n\n## YolosImageProcessor\n\n[[autodoc]] YolosImageProcessor\n    - preprocess\n    - pad\n    - post_process_object_detection\n\n## YolosFeatureExtractor\n\n[[autodoc]] YolosFeatureExtractor\n    - __call__\n    - pad\n    - post_process_object_detection\n\n## YolosModel\n\n[[autodoc]] YolosModel\n    - forward\n\n## YolosForObjectDetection\n\n[[autodoc]] YolosForObjectDetection\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ddd1a84a-e99e-454a-91bb-f3cd0aa2ef9a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ddd1a84a-e99e-454a-91bb-f3cd0aa2ef9a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"063e55a1-652b-49c4-a23b-82cf385e8184\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be408cf4-5cee-45f4-b946-61a8febbfdcf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bc343e37f3b930bfc8bd338c012550f5bcfac624bd82d5dc3bfbbc05d795dbd9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1898, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "ref_doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "chroma:document": "# Mistral\n\n## Overview\n\nMistral was introduced in the [this blogpost](https://mistral.ai/news/announcing-mistral-7b/) by Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, L\u00e9lio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth\u00e9e Lacroix, William El Sayed.\n\nThe introduction of the blog post says:\n\n*Mistral AI team is proud to release Mistral 7B, the most powerful language model for its size to date.*\n\nMistral-7B is the first large language model (LLM) released by [mistral.ai](https://mistral.ai/).\n\n### Architectural details\n\nMistral-7B is a decoder-only Transformer with the following architectural choices:\n\n- Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens\n- GQA (Grouped Query Attention) - allowing faster inference and lower cache size.\n- Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.\n\nFor more details refer to the [release blog post](https://mistral.ai/news/announcing-mistral-7b/).\n\n### License\n\n`Mistral-7B` is released under the Apache 2.0 license.\n\n## Usage tips\n\nThe Mistral team has released 3 checkpoints:\n\n- a base model, [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1), which has been pre-trained to predict the next token on internet-scale data.\n- an instruction tuned model, [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), which is the base model optimized for chat purposes using supervised fine-tuning (SFT) and direct preference optimization (DPO).\n- an improved instruction tuned model, [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), which improves upon v1."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: be408cf4-5cee-45f4-b946-61a8febbfdcf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"be408cf4-5cee-45f4-b946-61a8febbfdcf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"063e55a1-652b-49c4-a23b-82cf385e8184\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ddd1a84a-e99e-454a-91bb-f3cd0aa2ef9a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ccad838274504e80eeb8c72d6d980aa1e0dec9991647f34d432f1eeb45c5197e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ad44e9d9-c45b-4c1b-b1a2-5fa7e3c6ce2b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"acf4615c01abc181125911bd2f6ab64d9fc43d6e933825e54f032dd7ad2f5bcb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 438, \"end_char_idx\": 3176, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "ref_doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "chroma:document": "The introduction of the blog post says:\n\n*Mistral AI team is proud to release Mistral 7B, the most powerful language model for its size to date.*\n\nMistral-7B is the first large language model (LLM) released by [mistral.ai](https://mistral.ai/).\n\n### Architectural details\n\nMistral-7B is a decoder-only Transformer with the following architectural choices:\n\n- Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens\n- GQA (Grouped Query Attention) - allowing faster inference and lower cache size.\n- Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.\n\nFor more details refer to the [release blog post](https://mistral.ai/news/announcing-mistral-7b/).\n\n### License\n\n`Mistral-7B` is released under the Apache 2.0 license.\n\n## Usage tips\n\nThe Mistral team has released 3 checkpoints:\n\n- a base model, [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1), which has been pre-trained to predict the next token on internet-scale data.\n- an instruction tuned model, [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), which is the base model optimized for chat purposes using supervised fine-tuning (SFT) and direct preference optimization (DPO).\n- an improved instruction tuned model, [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), which improves upon v1.\n\nThe base model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"My favourite condiment is to ...\"\n```\n\nThe instruction tuned model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\")\n\n>>> messages = [\n...     {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ad44e9d9-c45b-4c1b-b1a2-5fa7e3c6ce2b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ad44e9d9-c45b-4c1b-b1a2-5fa7e3c6ce2b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"063e55a1-652b-49c4-a23b-82cf385e8184\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"be408cf4-5cee-45f4-b946-61a8febbfdcf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"861b71bdaa695c421eb82711931ab8b2f4610ca1caa2cb7d54d74ae6b23e6e26\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"663f72cc-cbde-4026-b1bb-cf54fabfd729\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f398234579368a2efad416727f2e3c29e45e8a0cf6fa257ed517f1d4d332f95\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1749, \"end_char_idx\": 4402, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "ref_doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "chroma:document": "- an improved instruction tuned model, [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), which improves upon v1.\n\nThe base model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"My favourite condiment is to ...\"\n```\n\nThe instruction tuned model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\")\n\n>>> messages = [\n...     {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"Mayonnaise can be made as follows: (...)\"\n```\n\nAs can be seen, the instruction-tuned model requires a [chat template](../chat_templating) to be applied to make sure the inputs are prepared in the right format.\n\n## Speeding up Mistral by using Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 663f72cc-cbde-4026-b1bb-cf54fabfd729
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"663f72cc-cbde-4026-b1bb-cf54fabfd729\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"063e55a1-652b-49c4-a23b-82cf385e8184\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ad44e9d9-c45b-4c1b-b1a2-5fa7e3c6ce2b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"31ed21d3e5ebb6607b889c0637e7f3c30580066a308a68838f79a15ae3275701\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1b09c81a-e0b8-41b5-8f48-b889405616b0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"46b909751e71ee9ddaf861cae077377f179f11f6fcf07f9b53ffe2866913b26b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2903, \"end_char_idx\": 6227, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "ref_doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "chroma:document": "\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"Mayonnaise can be made as follows: (...)\"\n```\n\nAs can be seen, the instruction-tuned model requires a [chat template](../chat_templating) to be applied to make sure the inputs are prepared in the right format.\n\n## Speeding up Mistral by using Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\nTo load and run a model using Flash Attention-2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"My favourite condiment is to (...)\"\n```\n\n### Expected speedups\n\nBelow is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `mistralai/Mistral-7B-v0.1` checkpoint and the Flash Attention 2 version of the model.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/mistral-7b-inference-large-seqlen.png\">\n</div>\n\n### Sliding window Attention\n\nThe current implementation supports the sliding window attention mechanism and memory efficient cache management. \nTo enable sliding window attention, just make sure to have a `flash-attn` version that is compatible with sliding window attention (`>=2.3.0`). \n\nThe Flash Attention-2 model uses also a more memory efficient cache slicing mechanism - as recommended per the official implementation of Mistral model that use rolling cache mechanism we keep the cache size fixed (`self.config.sliding_window`), support batched generation only for `padding_side=\"left\"` and use the absolute position of the current token to compute the positional embedding."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1b09c81a-e0b8-41b5-8f48-b889405616b0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1b09c81a-e0b8-41b5-8f48-b889405616b0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"063e55a1-652b-49c4-a23b-82cf385e8184\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"663f72cc-cbde-4026-b1bb-cf54fabfd729\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba58db1a70d3158c3c9cbe25f48c8b48bc301045762e59b44358aae7bf5c422d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"57bb7162-56d0-4aa2-abe2-09e5c31dc3a7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4741cf3d0aeffb72a425d375f5a6d9392811808703d34a151d6cbfdafaf5adc4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5379, \"end_char_idx\": 8620, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "ref_doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "chroma:document": "<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/mistral-7b-inference-large-seqlen.png\">\n</div>\n\n### Sliding window Attention\n\nThe current implementation supports the sliding window attention mechanism and memory efficient cache management. \nTo enable sliding window attention, just make sure to have a `flash-attn` version that is compatible with sliding window attention (`>=2.3.0`). \n\nThe Flash Attention-2 model uses also a more memory efficient cache slicing mechanism - as recommended per the official implementation of Mistral model that use rolling cache mechanism we keep the cache size fixed (`self.config.sliding_window`), support batched generation only for `padding_side=\"left\"` and use the absolute position of the current token to compute the positional embedding.\n\n## Shrinking down Mistral using quantization\n\nAs the Mistral model has 7 billion parameters, that would require about 14GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter),that requires only about 3.5GB of RAM.\n\nQuantizing a model is as simple as passing a `quantization_config` to the model. Below, we'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\n>>> # specify how to quantize the model\n>>> quantization_config = BitsAndBytesConfig(\n...         load_in_4bit=True,\n...         bnb_4bit_quant_type=\"nf4\",\n...         bnb_4bit_compute_dtype=\"torch.float16\",\n... )\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\", quantization_config=True, device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.2\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> messages = [\n...     {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"The expected output\"\n```\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .\nThe original code can be found [here](https://github.com/mistralai/mistral-src).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mistral. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 57bb7162-56d0-4aa2-abe2-09e5c31dc3a7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mistral", "title": "Mistral", "tokens": 2743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"57bb7162-56d0-4aa2-abe2-09e5c31dc3a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"063e55a1-652b-49c4-a23b-82cf385e8184\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9033034b170f4ce26227fa50c0302dc7886bc811f19c94f39cfe20d57a5a5d7c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1b09c81a-e0b8-41b5-8f48-b889405616b0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mistral\", \"title\": \"Mistral\", \"tokens\": 2743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8a0fffdba35c07f00f0d33f4de8075b576a6b03b02cfd2277d4d1d5ec4c3260b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7523, \"end_char_idx\": 10155, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "ref_doc_id": "063e55a1-652b-49c4-a23b-82cf385e8184", "chroma:document": "\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"The expected output\"\n```\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .\nThe original code can be found [here](https://github.com/mistralai/mistral-src).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mistral. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A demo notebook to perform supervised fine-tuning (SFT) of Mistral-7B can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Mistral/Supervised_fine_tuning_(SFT)_of_an_LLM_using_Hugging_Face_tooling.ipynb). \ud83c\udf0e\n- A [blog post](https://www.philschmid.de/fine-tune-llms-in-2024-with-trl) on how to fine-tune LLMs in 2024 using Hugging Face tooling. \ud83c\udf0e\n- The [Alignment Handbook](https://github.com/huggingface/alignment-handbook) by Hugging Face includes scripts and recipes to perform supervised fine-tuning (SFT) and direct preference optimization with Mistral-7B. This includes scripts for full fine-tuning, QLoRa on a single GPU as well as multi-GPU fine-tuning.\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## MistralConfig\n\n[[autodoc]] MistralConfig\n\n## MistralModel\n\n[[autodoc]] MistralModel\n    - forward\n\n## MistralForCausalLM\n\n[[autodoc]] MistralForCausalLM\n    - forward\n\n## MistralForSequenceClassification\n\n[[autodoc]] MistralForSequenceClassification\n    - forward\n\n## MistralForTokenClassification\n\n[[autodoc]] MistralForTokenClassification\n    - forward\n\n## FlaxMistralModel\n\n[[autodoc]] FlaxMistralModel\n    - __call__\n\n## FlaxMistralForCausalLM\n\n[[autodoc]] FlaxMistralForCausalLM\n    - __call__\n\n## TFMistralModel\n\n[[autodoc]] TFMistralModel\n    - call\n\n## TFMistralForCausalLM\n\n[[autodoc]] TFMistralForCausalLM\n    - call\n\n## TFMistralForSequenceClassification\n\n[[autodoc]] TFMistralForSequenceClassification\n    - call"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 57eb77f6-4fe8-4b53-9655-d37e0666805a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swin2sr", "title": "Swin2SR", "tokens": 850, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"57eb77f6-4fe8-4b53-9655-d37e0666805a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin2sr\", \"title\": \"Swin2SR\", \"tokens\": 850, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9b3134c6-039c-467f-b74c-bd3ed0c1c5a8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin2sr\", \"title\": \"Swin2SR\", \"tokens\": 850, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"343ec08353c1cb319d97cb30e003eac6d53690b2e306ee1d0e9ed6062b948745\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2878, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9b3134c6-039c-467f-b74c-bd3ed0c1c5a8", "doc_id": "9b3134c6-039c-467f-b74c-bd3ed0c1c5a8", "ref_doc_id": "9b3134c6-039c-467f-b74c-bd3ed0c1c5a8", "chroma:document": "# Swin2SR\n\n## Overview\n\nThe Swin2SR model was proposed in [Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration](https://arxiv.org/abs/2209.11345) by Marcos V. Conde, Ui-Jin Choi, Maxime Burchi, Radu Timofte.\nSwin2R improves the [SwinIR](https://github.com/JingyunLiang/SwinIR/) model by incorporating [Swin Transformer v2](swinv2) layers which mitigates issues such as training instability, resolution gaps between pre-training\nand fine-tuning, and hunger on data.\n\nThe abstract from the paper is the following:\n\n*Compression plays an important role on the efficient transmission and storage of images and videos through band-limited systems such as streaming services, virtual reality or videogames. However, compression unavoidably leads to artifacts and the loss of the original information, which may severely degrade the visual quality. For these reasons, quality enhancement of compressed images has become a popular research topic. While most state-of-the-art image restoration methods are based on convolutional neural networks, other transformers-based methods such as SwinIR, show impressive performance on these tasks.\nIn this paper, we explore the novel Swin Transformer V2, to improve SwinIR for image super-resolution, and in particular, the compressed input scenario. Using this method we can tackle the major issues in training transformer vision models, such as training instability, resolution gaps between pre-training and fine-tuning, and hunger on data. We conduct experiments on three representative tasks: JPEG compression artifacts removal, image super-resolution (classical and lightweight), and compressed image super-resolution. Experimental results demonstrate that our method, Swin2SR, can improve the training convergence and performance of SwinIR, and is a top-5 solution at the \"AIM 2022 Challenge on Super-Resolution of Compressed Image and Video\".*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/swin2sr_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Swin2SR architecture. Taken from the <a href=\"https://arxiv.org/abs/2209.11345\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/mv-lab/swin2sr).\n\n## Resources\n\nDemo notebooks for Swin2SR can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Swin2SR).\n\nA demo Space for image super-resolution with SwinSR can be found [here](https://huggingface.co/spaces/jjourney1125/swin2sr).\n\n## Swin2SRImageProcessor\n\n[[autodoc]] Swin2SRImageProcessor\n    - preprocess\n\n## Swin2SRConfig\n\n[[autodoc]] Swin2SRConfig\n\n## Swin2SRModel\n\n[[autodoc]] Swin2SRModel\n    - forward\n\n## Swin2SRForImageSuperResolution\n\n[[autodoc]] Swin2SRForImageSuperResolution\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 10a52268-6430-451a-8296-a12d20ab09fe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flava", "title": "FLAVA", "tokens": 777, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"10a52268-6430-451a-8296-a12d20ab09fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flava\", \"title\": \"FLAVA\", \"tokens\": 777, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ee595889-74b9-4be5-9834-a06f47347031\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flava\", \"title\": \"FLAVA\", \"tokens\": 777, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"50336de5b042fc0e07c287fe075c94818fd039cb0ecf2fd39b0e4cc488b2f61d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2373, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ee595889-74b9-4be5-9834-a06f47347031", "doc_id": "ee595889-74b9-4be5-9834-a06f47347031", "ref_doc_id": "ee595889-74b9-4be5-9834-a06f47347031", "chroma:document": "# FLAVA\n\n## Overview\n\nThe FLAVA model was proposed in [FLAVA: A Foundational Language And Vision Alignment Model](https://arxiv.org/abs/2112.04482) by Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela and is accepted at CVPR 2022.\n\nThe paper aims at creating a single unified foundation model which can work across vision, language\nas well as vision-and-language multimodal tasks.\n\nThe abstract from the paper is the following:\n\n*State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety\nof downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal\n(with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising\ndirection would be to use a single holistic universal model, as a \"foundation\", that targets all modalities\nat once -- a true vision and language foundation model should be good at vision tasks, language tasks, and\ncross- and multi-modal vision and language tasks. We introduce FLAVA as such a model and demonstrate\nimpressive performance on a wide range of 35 tasks spanning these target modalities.*\n\nThis model was contributed by [aps](https://huggingface.co/aps). The original code can be found [here](https://github.com/facebookresearch/multimodal/tree/main/examples/flava).\n\n## FlavaConfig\n\n[[autodoc]] FlavaConfig\n\n## FlavaTextConfig\n\n[[autodoc]] FlavaTextConfig\n\n## FlavaImageConfig\n\n[[autodoc]] FlavaImageConfig\n\n## FlavaMultimodalConfig\n\n[[autodoc]] FlavaMultimodalConfig\n\n## FlavaImageCodebookConfig\n\n[[autodoc]] FlavaImageCodebookConfig\n\n## FlavaProcessor\n\n[[autodoc]] FlavaProcessor\n\n## FlavaFeatureExtractor\n\n[[autodoc]] FlavaFeatureExtractor\n\n## FlavaImageProcessor\n\n[[autodoc]] FlavaImageProcessor\n    - preprocess\n\n## FlavaForPreTraining\n\n[[autodoc]] FlavaForPreTraining\n    - forward\n\n## FlavaModel\n\n[[autodoc]] FlavaModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## FlavaImageCodebook\n\n[[autodoc]] FlavaImageCodebook\n    - forward\n    - get_codebook_indices\n    - get_codebook_probs\n\n## FlavaTextModel\n\n[[autodoc]] FlavaTextModel\n    - forward\n\n## FlavaImageModel\n\n[[autodoc]] FlavaImageModel\n    - forward\n\n## FlavaMultimodalModel\n\n[[autodoc]] FlavaMultimodalModel\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ed8a442f-2ecd-4fa9-a178-762d86e524c9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava-next-video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ed8a442f-2ecd-4fa9-a178-762d86e524c9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0fa296e5-636d-4cfe-a5d0-cb8c478b04f4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"13494654b404183d13c590e5c518e468631adb53114fbba8583ba016b6f0a60b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6c04679c-2588-4ac0-b12d-132d347b6e36\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b1d4e05a9a2ea26c818f5437642aa636fdb448653924674a6de5fc6d7deffba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3383, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "ref_doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "chroma:document": "# LLaVa-NeXT-Video\n\n## Overview\n\nThe LLaVa-NeXT-Video model was proposed in [LLaVA-NeXT: A Strong Zero-shot Video Understanding Model\n](https://llava-vl.github.io/blog/2024-04-30-llava-next-video/) by Yuanhan Zhang, Bo Li, Haotian Liu, Yong Jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, Chunyuan Li. LLaVa-NeXT-Video improves upon [LLaVa-NeXT](llava_next) by fine-tuning on a mix if video and image dataset thus increasing the model's performance on videos.\n\n[LLaVA-NeXT](llava_next) surprisingly has strong performance in understanding video content in zero-shot fashion with the AnyRes technique that it uses. The AnyRes technique naturally represents a high-resolution image into multiple images. This technique is naturally generalizable to represent videos because videos can be considered as a set of frames (similar to a set of images in LLaVa-NeXT). The current version of LLaVA-NeXT makes use of AnyRes and trains with supervised fine-tuning (SFT) on top of LLaVA-Next on video data to achieves better video understanding capabilities.The model is a current SOTA among open-source models on [VideoMME bench](https://arxiv.org/abs/2405.21075).\n\n\nThe introduction from the blog is the following:\n\nOn January 30, 2024, we released LLaVA-NeXT, an open-source Large Multimodal Model (LMM) that has been trained exclusively on text-image data. With the proposed AnyRes technique, it boosts capabilities in reasoning, OCR, and world knowledge, demonstrating remarkable performance across a spectrum of image-based multimodal understanding tasks, and even exceeding Gemini-Pro on several image benchmarks, e.g. MMMU and MathVista.\n\n**In today\u2019s exploration, we delve into the performance of LLaVA-NeXT within the realm of video understanding tasks. We reveal that LLaVA-NeXT surprisingly has strong performance in understanding video content. The current version of LLaVA-NeXT for videos has several improvements:\n\n- Zero-shot video representation capabilities with AnyRes: The AnyRes technique naturally represents a high-resolution image into multiple images that a pre-trained VIT is able to digest, and forms them into a concantenated sequence. This technique is naturally generalizable to represent videos (consisting of multiple frames), allowing the image-only-trained LLaVA-Next model to perform surprisingly well on video tasks. Notably, this is the first time that LMMs show strong zero-shot modality transfer ability.\n- Inference with length generalization improves on longer videos. The linear scaling technique enables length generalization, allowing LLaVA-NeXT to effectively handle long-video beyond the limitation of the \"max_token_length\" of the LLM.\n- Strong video understanding ability. (1) LLaVA-Next-Image, which combines the above two techniques, yields superior zero-shot performance than open-source LMMs tuned on videos. (2) LLaVA-Next-Video, further supervised fine-tuning (SFT) LLaVA-Next-Image on video data, achieves better video understanding capabilities compared to LLaVA-Next-Image. (3) LLaVA-Next-Video-DPO, which aligns the model response with AI feedback using direct preference optimization (DPO), showing significant performance boost.\n- Efficient deployment and inference with SGLang. It allows 5x faster inference on video tasks, allowing more scalable serving such as million-level video re-captioning. See instructions in our repo.**"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6c04679c-2588-4ac0-b12d-132d347b6e36
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava-next-video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6c04679c-2588-4ac0-b12d-132d347b6e36\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0fa296e5-636d-4cfe-a5d0-cb8c478b04f4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"13494654b404183d13c590e5c518e468631adb53114fbba8583ba016b6f0a60b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ed8a442f-2ecd-4fa9-a178-762d86e524c9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d11aaab0ab1ec621e3fc23f6109a4b2dc5ae95749d18127848ca09644818bafa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dede948e-41bc-4ccc-a97f-07052eaa3791\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f2c09721d494f545b6e99cce1c4744f9df6c23e6a2c7840dc5f61df1168ce519\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3386, \"end_char_idx\": 6352, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "ref_doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "chroma:document": "This model was contributed by [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).\nThe original code can be found [here](https://github.com/LLaVA-VL/LLaVA-NeXT/tree/inference).\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n<Tip warning={true}>\n\n- Llava-Next uses different number of patches for images and thus has to pad the inputs inside modeling code, aside from the padding done when processing the inputs. The default setting is \"left-padding\" if model is in `eval()` mode, otherwise \"right-padding\".\n\n</Tip>\n\n\n- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use tokenizer's `apply_chat_template` to format your prompts correctly. Below is an example of how to do that.\n\nWe will use [LLaVA-NeXT-Video-7B-hf](https://huggingface.co/llava-hf/LLaVA-NeXT-Video-7B-hf) and a conversation history of videos and images. Each content field has to be a list of dicts, as follows:\n\n```python\nfrom transformers import LlavaNextVideoProcessor\n\nprocessor = LlavaNextVideoProcessor.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\")\n\nconversation = [\n    {\n        \"role\": \"system\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\"},\n            ],\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            {\"type\": \"image\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your visuals\nprint(text_prompt)\n```\n\n## Usage example\n\n### Single Media Mode\n\nThe model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):\n\n```python\nimport av\nimport torch\nimport numpy as np\nfrom transformers import LlavaNextVideoForConditionalGeneration, LlavaNextVideoProcessor\n\ndef read_video_pyav(container, indices):\n    '''\n    Decode the video with PyAV decoder.\n    Args:\n        container (`av.container.input.InputContainer`): PyAV container.\n        indices (`List[int]`): List of frame indices to decode.\n    Returns:\n        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).\n    '''"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: dede948e-41bc-4ccc-a97f-07052eaa3791
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava-next-video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dede948e-41bc-4ccc-a97f-07052eaa3791\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0fa296e5-636d-4cfe-a5d0-cb8c478b04f4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"13494654b404183d13c590e5c518e468631adb53114fbba8583ba016b6f0a60b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6c04679c-2588-4ac0-b12d-132d347b6e36\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0c6ab1735910f112abdf881e29f820e01c1a6bd6b1e4642c49bc41f766422d39\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bd6f8ba7-d9ef-4d01-aacc-fe4f9c4397ea\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"517c920fa1528ae0a23a983e7788761f55a82254c5980e254af75eaa75e746a0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4870, \"end_char_idx\": 7859, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "ref_doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "chroma:document": "The assistant gives helpful, detailed, and polite answers to the human's questions.\"},\n            ],\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            {\"type\": \"image\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your visuals\nprint(text_prompt)\n```\n\n## Usage example\n\n### Single Media Mode\n\nThe model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):\n\n```python\nimport av\nimport torch\nimport numpy as np\nfrom transformers import LlavaNextVideoForConditionalGeneration, LlavaNextVideoProcessor\n\ndef read_video_pyav(container, indices):\n    '''\n    Decode the video with PyAV decoder.\n    Args:\n        container (`av.container.input.InputContainer`): PyAV container.\n        indices (`List[int]`): List of frame indices to decode.\n    Returns:\n        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).\n    '''\n    frames = []\n    container.seek(0)\n    start_index = indices[0]\n    end_index = indices[-1]\n    for i, frame in enumerate(container.decode(video=0)):\n        if i > end_index:\n            break\n        if i >= start_index and i in indices:\n            frames.append(frame)\n    return np.stack([x.to_ndarray(format=\"rgb24\") for x in frames])\n\n# Load the model in half-precision\nmodel = LlavaNextVideoForConditionalGeneration.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\", torch_dtype=torch.float16, device_map=\"auto\")\nprocessor = LlavaNextVideoProcessor.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\")\n\n# Load the video as an np.array, sampling uniformly 8 frames (can sample more for longer videos)\nvideo_path = hf_hub_download(repo_id=\"raushan-testing-hf/videos-test\", filename=\"sample_demo_1.mp4\", repo_type=\"dataset\")\ncontainer = av.open(video_path)\ntotal_frames = container.streams.video[0].frames\nindices = np.arange(0, total_frames, total_frames / 8).astype(int)\nvideo = read_video_pyav(container, indices)\n\nconversation = [\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\n\nprompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\ninputs = processor(text=prompt, videos=video, return_tensors=\"pt\")\n\nout = model.generate(**inputs, max_new_tokens=60)\nprocessor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: bd6f8ba7-d9ef-4d01-aacc-fe4f9c4397ea
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava-next-video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bd6f8ba7-d9ef-4d01-aacc-fe4f9c4397ea\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0fa296e5-636d-4cfe-a5d0-cb8c478b04f4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"13494654b404183d13c590e5c518e468631adb53114fbba8583ba016b6f0a60b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dede948e-41bc-4ccc-a97f-07052eaa3791\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bc075d6a807b21b23b46b0feea83fc2177cb274f96c396a17384b5b8175cc3b7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ab8d17e-2fa6-435d-b755-cc0aaced6dcd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e71dcaf6b64506d99358b9130c6a60b4d1923cf91d4323e535ff36c65a5ea36\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6357, \"end_char_idx\": 9804, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "ref_doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "chroma:document": "frames = []\n    container.seek(0)\n    start_index = indices[0]\n    end_index = indices[-1]\n    for i, frame in enumerate(container.decode(video=0)):\n        if i > end_index:\n            break\n        if i >= start_index and i in indices:\n            frames.append(frame)\n    return np.stack([x.to_ndarray(format=\"rgb24\") for x in frames])\n\n# Load the model in half-precision\nmodel = LlavaNextVideoForConditionalGeneration.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\", torch_dtype=torch.float16, device_map=\"auto\")\nprocessor = LlavaNextVideoProcessor.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\")\n\n# Load the video as an np.array, sampling uniformly 8 frames (can sample more for longer videos)\nvideo_path = hf_hub_download(repo_id=\"raushan-testing-hf/videos-test\", filename=\"sample_demo_1.mp4\", repo_type=\"dataset\")\ncontainer = av.open(video_path)\ntotal_frames = container.streams.video[0].frames\nindices = np.arange(0, total_frames, total_frames / 8).astype(int)\nvideo = read_video_pyav(container, indices)\n\nconversation = [\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\n\nprompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\ninputs = processor(text=prompt, videos=video, return_tensors=\"pt\")\n\nout = model.generate(**inputs, max_new_tokens=60)\nprocessor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n```\n\n\n### Mixed Media Mode\n\nThe model can also generate from an interleaved image-video inputs. However note, that it was not trained in interleaved image-video setting which might affect the performance. Below is an example usage for mixed media input, add the following lines to the above code snippet: \n\n```python\nfrom PIL import Image\nimport requests\n\n# Generate from image and video mixed inputs\n# Load and image and write a new prompt\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nconversation = [\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"How many cats are there in the image?\"},\n            {\"type\": \"image\"},\n            ],\n    },\n    {\n\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"There are two cats\"}],\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\nprompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\ninputs = processor(text=prompt, images=image, videos=clip, padding=True, return_tensors=\"pt\")\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_length=50)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes for memory efficiency\n\nThe model can be loaded in lower bits, significantly reducing memory burden while maintaining the performance of the original model. This allows for efficient deployment on resource-constrained cases. \n\nFirst make sure to install bitsandbytes by running `pip install bitsandbytes` and to have access to a CUDA compatible GPU device. Load the quantized model by simply adding [`BitsAndBytesConfig`](../main_classes/quantization#transformers.BitsAndBytesConfig) as shown below:"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4ab8d17e-2fa6-435d-b755-cc0aaced6dcd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava-next-video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4ab8d17e-2fa6-435d-b755-cc0aaced6dcd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0fa296e5-636d-4cfe-a5d0-cb8c478b04f4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"13494654b404183d13c590e5c518e468631adb53114fbba8583ba016b6f0a60b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bd6f8ba7-d9ef-4d01-aacc-fe4f9c4397ea\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava-next-video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c934dd328c6f0c8e5d7e0c062bacb2ae3dd9e38b2647a046d2045c231b919f1c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9807, \"end_char_idx\": 11655, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "ref_doc_id": "0fa296e5-636d-4cfe-a5d0-cb8c478b04f4", "chroma:document": "```python\nfrom transformers import LlavaNextVideoForConditionalGeneration, LlavaNextVideoProcessor\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = LlavaNextVideoForConditionalGeneration.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\", quantization_config=quantization_config, device_map=\"auto\")\n```\n\n\n### Flash-Attention 2 to speed-up generation\n\nAdditionally, we can greatly speed-up model inference by using [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nAlso, you should have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.\n\nTo load and run a model using Flash Attention-2, simply add `attn_implementation=\"flash_attention_2\"` when loading the model as follows:\n\n```python\nfrom transformers import LlavaNextVideoForConditionalGeneration\n\nmodel = LlavaNextVideoForConditionalGeneration.from_pretrained(\n    \"llava-hf/LLaVA-NeXT-Video-7B-hf\", \n    torch_dtype=torch.float16, \n    attn_implementation=\"flash_attention_2\",\n).to(0)\n```\n\n\n\n## LlavaNextVideoConfig\n\n[[autodoc]] LlavaNextVideoConfig\n\n## LlavaNextVideoProcessor\n\n[[autodoc]] LlavaNextVideoProcessor\n\n## LlavaNextVideoImageProcessor\n\n[[autodoc]] LlavaNextVideoImageProcessor\n\n## LlavaNextVideoForConditionalGeneration\n\n[[autodoc]] LlavaNextVideoForConditionalGeneration\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 409cd068-e77f-4100-8f53-3a1fd23e56c0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinat", "title": "Dilated Neighborhood Attention Transformer", "tokens": 1172, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"409cd068-e77f-4100-8f53-3a1fd23e56c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e45b753e-6fd6-4eaf-b35a-3ed258c62bb6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a1072cbe1f2b645db94485c34c83e539ad487047e286b4cc53bdadc1154c23f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3edf2d34-8b59-4d13-b222-41debfb1c6d8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fcd5672203e28fa6856a2a6a1d30c5a639f8e497b01722aa4528c3edffb35d13\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3415, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e45b753e-6fd6-4eaf-b35a-3ed258c62bb6", "doc_id": "e45b753e-6fd6-4eaf-b35a-3ed258c62bb6", "ref_doc_id": "e45b753e-6fd6-4eaf-b35a-3ed258c62bb6", "chroma:document": "# Dilated Neighborhood Attention Transformer\n\n## Overview\n\nDiNAT was proposed in [Dilated Neighborhood Attention Transformer](https://arxiv.org/abs/2209.15001)\nby Ali Hassani and Humphrey Shi.\n\nIt extends [NAT](nat) by adding a Dilated Neighborhood Attention pattern to capture global context,\nand shows significant performance improvements over it.\n\nThe abstract from the paper is the following:\n\n*Transformers are quickly becoming one of the most heavily applied deep learning architectures across modalities,\ndomains, and tasks. In vision, on top of ongoing efforts into plain transformers, hierarchical transformers have\nalso gained significant attention, thanks to their performance and easy integration into existing frameworks.\nThese models typically employ localized attention mechanisms, such as the sliding-window Neighborhood Attention (NA)\nor Swin Transformer's Shifted Window Self Attention. While effective at reducing self attention's quadratic complexity,\nlocal attention weakens two of the most desirable properties of self attention: long range inter-dependency modeling,\nand global receptive field. In this paper, we introduce Dilated Neighborhood Attention (DiNA), a natural, flexible and\nefficient extension to NA that can capture more global context and expand receptive fields exponentially at no\nadditional cost. NA's local attention and DiNA's sparse global attention complement each other, and therefore we\nintroduce Dilated Neighborhood Attention Transformer (DiNAT), a new hierarchical vision transformer built upon both.\nDiNAT variants enjoy significant improvements over strong baselines such as NAT, Swin, and ConvNeXt.\nOur large model is faster and ahead of its Swin counterpart by 1.5% box AP in COCO object detection,\n1.3% mask AP in COCO instance segmentation, and 1.1% mIoU in ADE20K semantic segmentation.\nPaired with new frameworks, our large variant is the new state of the art panoptic segmentation model on COCO (58.2 PQ)\nand ADE20K (48.5 PQ), and instance segmentation model on Cityscapes (44.5 AP) and ADE20K (35.4 AP) (no extra data).\nIt also matches the state of the art specialized semantic segmentation models on ADE20K (58.2 mIoU),\nand ranks second on Cityscapes (84.5 mIoU) (no extra data). *\n\n<img\nsrc=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dilated-neighborhood-attention-pattern.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Neighborhood Attention with different dilation values.\nTaken from the <a href=\"https://arxiv.org/abs/2209.15001\">original paper</a>.</small>\n\nThis model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).\nThe original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).\n\n## Usage tips\n\nDiNAT can be used as a *backbone*. When `output_hidden_states = True`,\nit will output both `hidden_states` and `reshaped_hidden_states`. The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than `(batch_size, height, width, num_channels)`.\n\nNotes:\n- DiNAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention and Dilated Neighborhood Attention.\nYou can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten), or build on your system by running `pip install natten`.\nNote that the latter will likely take time to compile."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3edf2d34-8b59-4d13-b222-41debfb1c6d8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinat", "title": "Dilated Neighborhood Attention Transformer", "tokens": 1172, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3edf2d34-8b59-4d13-b222-41debfb1c6d8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e45b753e-6fd6-4eaf-b35a-3ed258c62bb6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a1072cbe1f2b645db94485c34c83e539ad487047e286b4cc53bdadc1154c23f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"409cd068-e77f-4100-8f53-3a1fd23e56c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinat\", \"title\": \"Dilated Neighborhood Attention Transformer\", \"tokens\": 1172, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a2bc34ee053c07572f21fd2ad1a957e74d71c53ae44c4f6502c86587768e7102\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2079, \"end_char_idx\": 4461, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e45b753e-6fd6-4eaf-b35a-3ed258c62bb6", "doc_id": "e45b753e-6fd6-4eaf-b35a-3ed258c62bb6", "ref_doc_id": "e45b753e-6fd6-4eaf-b35a-3ed258c62bb6", "chroma:document": "It also matches the state of the art specialized semantic segmentation models on ADE20K (58.2 mIoU),\nand ranks second on Cityscapes (84.5 mIoU) (no extra data). *\n\n<img\nsrc=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dilated-neighborhood-attention-pattern.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Neighborhood Attention with different dilation values.\nTaken from the <a href=\"https://arxiv.org/abs/2209.15001\">original paper</a>.</small>\n\nThis model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).\nThe original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).\n\n## Usage tips\n\nDiNAT can be used as a *backbone*. When `output_hidden_states = True`,\nit will output both `hidden_states` and `reshaped_hidden_states`. The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than `(batch_size, height, width, num_channels)`.\n\nNotes:\n- DiNAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention and Dilated Neighborhood Attention.\nYou can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten), or build on your system by running `pip install natten`.\nNote that the latter will likely take time to compile. NATTEN does not support Windows devices yet.\n- Patch size of 4 is only supported at the moment.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DiNAT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`DinatForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DinatConfig\n\n[[autodoc]] DinatConfig\n\n## DinatModel\n\n[[autodoc]] DinatModel\n    - forward\n\n## DinatForImageClassification\n\n[[autodoc]] DinatForImageClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: fba0b918-eaed-4f91-85af-9b96785ff166
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-conformer", "title": "Wav2Vec2-Conformer", "tokens": 922, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fba0b918-eaed-4f91-85af-9b96785ff166\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-conformer\", \"title\": \"Wav2Vec2-Conformer\", \"tokens\": 922, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4fc2d19c-6e8b-4e3e-b58a-0755c151cc96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-conformer\", \"title\": \"Wav2Vec2-Conformer\", \"tokens\": 922, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d265c77f0218a4c660aeb6d8597bdb066e5a6a889536ec3a83dc80ee41d44c68\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2653, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4fc2d19c-6e8b-4e3e-b58a-0755c151cc96", "doc_id": "4fc2d19c-6e8b-4e3e-b58a-0755c151cc96", "ref_doc_id": "4fc2d19c-6e8b-4e3e-b58a-0755c151cc96", "chroma:document": "# Wav2Vec2-Conformer\n\n## Overview\n\nThe Wav2Vec2-Conformer was added to an updated version of [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://arxiv.org/abs/2010.05171) by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Sravya Popuri, Dmytro Okhonko, Juan Pino.\n\nThe official results of the model can be found in Table 3 and Table 4 of the paper.\n\nThe Wav2Vec2-Conformer weights were released by the Meta AI team within the [Fairseq library](https://github.com/pytorch/fairseq/blob/main/examples/wav2vec/README.md#pre-trained-models).\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).\nThe original code can be found [here](https://github.com/pytorch/fairseq/tree/main/examples/wav2vec).\n\nNote: Meta (FAIR) released a new version of [Wav2Vec2-BERT 2.0](https://huggingface.co/docs/transformers/en/model_doc/wav2vec2-bert) - it's pretrained on 4.5M hours of audio. We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).\n\n## Usage tips\n\n- Wav2Vec2-Conformer follows the same architecture as Wav2Vec2, but replaces the *Attention*-block with a *Conformer*-block\n  as introduced in [Conformer: Convolution-augmented Transformer for Speech Recognition](https://arxiv.org/abs/2005.08100).\n- For the same number of layers, Wav2Vec2-Conformer requires more parameters than Wav2Vec2, but also yields \nan improved word error rate.\n- Wav2Vec2-Conformer uses the same tokenizer and feature extractor as Wav2Vec2.\n- Wav2Vec2-Conformer can use either no relative position embeddings, Transformer-XL-like position embeddings, or\n  rotary position embeddings by setting the correct `config.position_embeddings_type`.\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## Wav2Vec2ConformerConfig\n\n[[autodoc]] Wav2Vec2ConformerConfig\n\n## Wav2Vec2Conformer specific outputs\n\n[[autodoc]] models.wav2vec2_conformer.modeling_wav2vec2_conformer.Wav2Vec2ConformerForPreTrainingOutput\n\n## Wav2Vec2ConformerModel\n\n[[autodoc]] Wav2Vec2ConformerModel\n    - forward\n\n## Wav2Vec2ConformerForCTC\n\n[[autodoc]] Wav2Vec2ConformerForCTC\n    - forward\n\n## Wav2Vec2ConformerForSequenceClassification\n\n[[autodoc]] Wav2Vec2ConformerForSequenceClassification\n    - forward\n\n## Wav2Vec2ConformerForAudioFrameClassification\n\n[[autodoc]] Wav2Vec2ConformerForAudioFrameClassification\n    - forward\n\n## Wav2Vec2ConformerForXVector\n\n[[autodoc]] Wav2Vec2ConformerForXVector\n    - forward\n\n## Wav2Vec2ConformerForPreTraining\n\n[[autodoc]] Wav2Vec2ConformerForPreTraining\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7e54edd9-6e7b-41c3-bfcf-2009335455df
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7e54edd9-6e7b-41c3-bfcf-2009335455df\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f7c5754e-633d-45f5-88cf-39e839e4efbe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d9989dd2-16f8-4257-af42-9748d8a5d3a1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2f19aa9b5a565347a92a30d9309605ab341a4bb34e1004d3d4acab511403a1d8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3308, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "ref_doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "chroma:document": "# MarkupLM\n\n## Overview\n\nThe MarkupLM model was proposed in [MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document\nUnderstanding](https://arxiv.org/abs/2110.08518) by Junlong Li, Yiheng Xu, Lei Cui, Furu Wei. MarkupLM is BERT, but\napplied to HTML pages instead of raw text documents. The model incorporates additional embedding layers to improve\nperformance, similar to [LayoutLM](layoutlm).\n\nThe model can be used for tasks like question answering on web pages or information extraction from web pages. It obtains\nstate-of-the-art results on 2 important benchmarks:\n- [WebSRC](https://x-lance.github.io/WebSRC/), a dataset for Web-Based Structural Reading Comprehension (a bit like SQuAD but for web pages)\n- [SWDE](https://www.researchgate.net/publication/221299838_From_one_tree_to_a_forest_a_unified_solution_for_structured_web_data_extraction), a dataset\nfor information extraction from web pages (basically named-entity recognition on web pages)\n\nThe abstract from the paper is the following:\n\n*Multimodal pre-training with text, layout, and image has made significant progress for Visually-rich Document\nUnderstanding (VrDU), especially the fixed-layout documents such as scanned document images. While, there are still a\nlarge number of digital documents where the layout information is not fixed and needs to be interactively and\ndynamically rendered for visualization, making existing layout-based pre-training approaches not easy to apply. In this\npaper, we propose MarkupLM for document understanding tasks with markup languages as the backbone such as\nHTML/XML-based documents, where text and markup information is jointly pre-trained. Experiment results show that the\npre-trained MarkupLM significantly outperforms the existing strong baseline models on several document understanding\ntasks. The pre-trained model and code will be publicly available.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/markuplm).\n\n## Usage tips\n\n- In addition to `input_ids`, [`~MarkupLMModel.forward`] expects 2 additional inputs, namely `xpath_tags_seq` and `xpath_subs_seq`.\nThese are the XPATH tags and subscripts respectively for each token in the input sequence.\n- One can use [`MarkupLMProcessor`] to prepare all data for the model. Refer to the [usage guide](#usage-markuplmprocessor) for more info.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/markuplm_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> MarkupLM architecture. Taken from the <a href=\"https://arxiv.org/abs/2110.08518\">original paper.</a> </small>\n\n## Usage: MarkupLMProcessor\n\nThe easiest way to prepare data for the model is to use [`MarkupLMProcessor`], which internally combines a feature extractor\n([`MarkupLMFeatureExtractor`]) and a tokenizer ([`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`]). The feature extractor is\nused to extract all nodes and xpaths from the HTML strings, which are then provided to the tokenizer, which turns them into the\ntoken-level inputs of the model (`input_ids` etc.). Note that you can still use the feature extractor and tokenizer separately,\nif you only want to handle one of the two tasks."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d9989dd2-16f8-4257-af42-9748d8a5d3a1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d9989dd2-16f8-4257-af42-9748d8a5d3a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f7c5754e-633d-45f5-88cf-39e839e4efbe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7e54edd9-6e7b-41c3-bfcf-2009335455df\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e65c5a58947af6692272e8de14f652cac694725f2a1a2ea8c1e2c4b65927753a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ed023711-f115-4268-9753-cf12508ec44b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5c81aac119072dcaf171a4d4a7cc2bc350f20bfd72f3dd1e9b4b4d9af9d25b3f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1673, \"end_char_idx\": 4990, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "ref_doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "chroma:document": "Experiment results show that the\npre-trained MarkupLM significantly outperforms the existing strong baseline models on several document understanding\ntasks. The pre-trained model and code will be publicly available.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/markuplm).\n\n## Usage tips\n\n- In addition to `input_ids`, [`~MarkupLMModel.forward`] expects 2 additional inputs, namely `xpath_tags_seq` and `xpath_subs_seq`.\nThese are the XPATH tags and subscripts respectively for each token in the input sequence.\n- One can use [`MarkupLMProcessor`] to prepare all data for the model. Refer to the [usage guide](#usage-markuplmprocessor) for more info.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/markuplm_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> MarkupLM architecture. Taken from the <a href=\"https://arxiv.org/abs/2110.08518\">original paper.</a> </small>\n\n## Usage: MarkupLMProcessor\n\nThe easiest way to prepare data for the model is to use [`MarkupLMProcessor`], which internally combines a feature extractor\n([`MarkupLMFeatureExtractor`]) and a tokenizer ([`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`]). The feature extractor is\nused to extract all nodes and xpaths from the HTML strings, which are then provided to the tokenizer, which turns them into the\ntoken-level inputs of the model (`input_ids` etc.). Note that you can still use the feature extractor and tokenizer separately,\nif you only want to handle one of the two tasks.\n\n```python\nfrom transformers import MarkupLMFeatureExtractor, MarkupLMTokenizerFast, MarkupLMProcessor\n\nfeature_extractor = MarkupLMFeatureExtractor()\ntokenizer = MarkupLMTokenizerFast.from_pretrained(\"microsoft/markuplm-base\")\nprocessor = MarkupLMProcessor(feature_extractor, tokenizer)\n```\n\nIn short, one can provide HTML strings (and possibly additional data) to [`MarkupLMProcessor`],\nand it will create the inputs expected by the model. Internally, the processor first uses\n[`MarkupLMFeatureExtractor`] to get a list of nodes and corresponding xpaths. The nodes and\nxpaths are then provided to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`], which converts them\nto token-level `input_ids`, `attention_mask`, `token_type_ids`, `xpath_subs_seq`, `xpath_tags_seq`.\nOptionally, one can provide node labels to the processor, which are turned into token-level `labels`.\n\n[`MarkupLMFeatureExtractor`] uses [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/), a Python library for\npulling data out of HTML and XML files, under the hood. Note that you can still use your own parsing solution of\nchoice, and provide the nodes and xpaths yourself to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`].\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: web page classification (training, inference) + token classification (inference), parse_html = True**\n\nThis is the simplest case, in which the processor will use the feature extractor to get all nodes and xpaths from the HTML."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ed023711-f115-4268-9753-cf12508ec44b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ed023711-f115-4268-9753-cf12508ec44b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f7c5754e-633d-45f5-88cf-39e839e4efbe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d9989dd2-16f8-4257-af42-9748d8a5d3a1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7e9d808b4a5e9fe746816e01ff850cb61a845b24397bcf49221a3b490cabb322\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"97ed75da-935e-4b34-bd85-86ef32699fb0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"10325888c18faf8589a4ab989b9d9c46832dd55801ce969ae171ac90b060144d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3310, \"end_char_idx\": 5976, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "ref_doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "chroma:document": "```python\nfrom transformers import MarkupLMFeatureExtractor, MarkupLMTokenizerFast, MarkupLMProcessor\n\nfeature_extractor = MarkupLMFeatureExtractor()\ntokenizer = MarkupLMTokenizerFast.from_pretrained(\"microsoft/markuplm-base\")\nprocessor = MarkupLMProcessor(feature_extractor, tokenizer)\n```\n\nIn short, one can provide HTML strings (and possibly additional data) to [`MarkupLMProcessor`],\nand it will create the inputs expected by the model. Internally, the processor first uses\n[`MarkupLMFeatureExtractor`] to get a list of nodes and corresponding xpaths. The nodes and\nxpaths are then provided to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`], which converts them\nto token-level `input_ids`, `attention_mask`, `token_type_ids`, `xpath_subs_seq`, `xpath_tags_seq`.\nOptionally, one can provide node labels to the processor, which are turned into token-level `labels`.\n\n[`MarkupLMFeatureExtractor`] uses [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/), a Python library for\npulling data out of HTML and XML files, under the hood. Note that you can still use your own parsing solution of\nchoice, and provide the nodes and xpaths yourself to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`].\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: web page classification (training, inference) + token classification (inference), parse_html = True**\n\nThis is the simplest case, in which the processor will use the feature extractor to get all nodes and xpaths from the HTML.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n\n>>> html_string = \"\"\"\n...  <!DOCTYPE html>\n...  <html>\n...  <head>\n...  <title>Hello world</title>\n...  </head>\n...  <body>\n...  <h1>Welcome</h1>\n...  <p>Here is my website.</p>\n...  </body>\n...  </html>\"\"\"\n\n>>> # note that you can also add provide all tokenizer parameters here such as padding, truncation\n>>> encoding = processor(html_string, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 2: web page classification (training, inference) + token classification (inference), parse_html=False**\n\nIn case one already has obtained all nodes and xpaths, one doesn't need the feature extractor. In that case, one should\nprovide the nodes and corresponding xpaths themselves to the processor, and make sure to set `parse_html` to `False`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 97ed75da-935e-4b34-bd85-86ef32699fb0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"97ed75da-935e-4b34-bd85-86ef32699fb0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f7c5754e-633d-45f5-88cf-39e839e4efbe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ed023711-f115-4268-9753-cf12508ec44b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e1471c3083f5d3bde961e8de6da27fb421d6b915c03802c2da511d3eff0c354d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5a09963f-f990-4e80-ba2a-8799c79be968\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"831c2b10a3bfaf1647354f7005aca8c680dd078f16e418a3a45ae510cb85e431\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4368, \"end_char_idx\": 7109, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "ref_doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "chroma:document": "Note that you can still use your own parsing solution of\nchoice, and provide the nodes and xpaths yourself to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`].\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: web page classification (training, inference) + token classification (inference), parse_html = True**\n\nThis is the simplest case, in which the processor will use the feature extractor to get all nodes and xpaths from the HTML.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n\n>>> html_string = \"\"\"\n...  <!DOCTYPE html>\n...  <html>\n...  <head>\n...  <title>Hello world</title>\n...  </head>\n...  <body>\n...  <h1>Welcome</h1>\n...  <p>Here is my website.</p>\n...  </body>\n...  </html>\"\"\"\n\n>>> # note that you can also add provide all tokenizer parameters here such as padding, truncation\n>>> encoding = processor(html_string, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 2: web page classification (training, inference) + token classification (inference), parse_html=False**\n\nIn case one already has obtained all nodes and xpaths, one doesn't need the feature extractor. In that case, one should\nprovide the nodes and corresponding xpaths themselves to the processor, and make sure to set `parse_html` to `False`.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n>>> processor.parse_html = False\n\n>>> nodes = [\"hello\", \"world\", \"how\", \"are\"]\n>>> xpaths = [\"/html/body/div/li[1]/div/span\", \"/html/body/div/li[1]/div/span\", \"html/body\", \"html/body/div\"]\n>>> encoding = processor(nodes=nodes, xpaths=xpaths, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 3: token classification (training), parse_html=False**\n\nFor token classification tasks (such as [SWDE](https://paperswithcode.com/dataset/swde)), one can also provide the\ncorresponding node labels in order to train a model. The processor will then convert these into token-level `labels`.\nBy default, it will only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the\n`ignore_index` of PyTorch's CrossEntropyLoss. In case you want all wordpieces of a word to be labeled, you can\ninitialize the tokenizer with `only_label_first_subword` set to `False`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5a09963f-f990-4e80-ba2a-8799c79be968
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5a09963f-f990-4e80-ba2a-8799c79be968\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f7c5754e-633d-45f5-88cf-39e839e4efbe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"97ed75da-935e-4b34-bd85-86ef32699fb0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2e6128931e15ae9a49d90834a094ff3245f9b4228ea78dd66690ff2551e64358\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5277f636-ed46-4963-bdb4-15985df2c1ab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"290e33ba402886c42f647e0463c70ed22fde44acddf7584bcea62ab87ac4e097\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5834, \"end_char_idx\": 8904, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "ref_doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "chroma:document": "In that case, one should\nprovide the nodes and corresponding xpaths themselves to the processor, and make sure to set `parse_html` to `False`.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n>>> processor.parse_html = False\n\n>>> nodes = [\"hello\", \"world\", \"how\", \"are\"]\n>>> xpaths = [\"/html/body/div/li[1]/div/span\", \"/html/body/div/li[1]/div/span\", \"html/body\", \"html/body/div\"]\n>>> encoding = processor(nodes=nodes, xpaths=xpaths, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 3: token classification (training), parse_html=False**\n\nFor token classification tasks (such as [SWDE](https://paperswithcode.com/dataset/swde)), one can also provide the\ncorresponding node labels in order to train a model. The processor will then convert these into token-level `labels`.\nBy default, it will only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the\n`ignore_index` of PyTorch's CrossEntropyLoss. In case you want all wordpieces of a word to be labeled, you can\ninitialize the tokenizer with `only_label_first_subword` set to `False`.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n>>> processor.parse_html = False\n\n>>> nodes = [\"hello\", \"world\", \"how\", \"are\"]\n>>> xpaths = [\"/html/body/div/li[1]/div/span\", \"/html/body/div/li[1]/div/span\", \"html/body\", \"html/body/div\"]\n>>> node_labels = [1, 2, 2, 1]\n>>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq', 'labels'])\n```\n\n**Use case 4: web page question answering (inference), parse_html=True**\n\nFor question answering tasks on web pages, you can provide a question to the processor. By default, the\nprocessor will use the feature extractor to get all nodes and xpaths, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n\n>>> html_string = \"\"\"\n...  <!DOCTYPE html>\n...  <html>\n...  <head>\n...  <title>Hello world</title>\n...  </head>\n...  <body>\n...  <h1>Welcome</h1>\n...  <p>My name is Niels.</p>\n...  </body>\n...  </html>\"\"\"\n\n>>> question = \"What's his name?\"\n>>> encoding = processor(html_string, questions=question, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 5: web page question answering (inference), parse_html=False**\n\nFor question answering tasks (such as WebSRC), you can provide a question to the processor. If you have extracted\nall nodes and xpaths yourself, you can provide them directly to the processor. Make sure to set `parse_html` to `False`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5277f636-ed46-4963-bdb4-15985df2c1ab
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5277f636-ed46-4963-bdb4-15985df2c1ab\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f7c5754e-633d-45f5-88cf-39e839e4efbe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5a09963f-f990-4e80-ba2a-8799c79be968\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/markuplm\", \"title\": \"MarkupLM\", \"tokens\": 2673, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1aaeaed103b7a8bf69e1de3effa3cb4c785e28dad0322427a259417cde01dfdf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7866, \"end_char_idx\": 10598, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "ref_doc_id": "f7c5754e-633d-45f5-88cf-39e839e4efbe", "chroma:document": "By default, the\nprocessor will use the feature extractor to get all nodes and xpaths, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n\n>>> html_string = \"\"\"\n...  <!DOCTYPE html>\n...  <html>\n...  <head>\n...  <title>Hello world</title>\n...  </head>\n...  <body>\n...  <h1>Welcome</h1>\n...  <p>My name is Niels.</p>\n...  </body>\n...  </html>\"\"\"\n\n>>> question = \"What's his name?\"\n>>> encoding = processor(html_string, questions=question, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n**Use case 5: web page question answering (inference), parse_html=False**\n\nFor question answering tasks (such as WebSRC), you can provide a question to the processor. If you have extracted\nall nodes and xpaths yourself, you can provide them directly to the processor. Make sure to set `parse_html` to `False`.\n\n```python\n>>> from transformers import MarkupLMProcessor\n\n>>> processor = MarkupLMProcessor.from_pretrained(\"microsoft/markuplm-base\")\n>>> processor.parse_html = False\n\n>>> nodes = [\"hello\", \"world\", \"how\", \"are\"]\n>>> xpaths = [\"/html/body/div/li[1]/div/span\", \"/html/body/div/li[1]/div/span\", \"html/body\", \"html/body/div\"]\n>>> question = \"What's his name?\"\n>>> encoding = processor(nodes=nodes, xpaths=xpaths, questions=question, return_tensors=\"pt\")\n>>> print(encoding.keys())\ndict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])\n```\n\n## Resources\n\n- [Demo notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/MarkupLM)\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n\n## MarkupLMConfig\n\n[[autodoc]] MarkupLMConfig\n    - all\n\n## MarkupLMFeatureExtractor\n\n[[autodoc]] MarkupLMFeatureExtractor\n    - __call__\n\n## MarkupLMTokenizer\n\n[[autodoc]] MarkupLMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## MarkupLMTokenizerFast\n\n[[autodoc]] MarkupLMTokenizerFast\n    - all\n\n## MarkupLMProcessor\n\n[[autodoc]] MarkupLMProcessor\n    - __call__\n\n## MarkupLMModel\n\n[[autodoc]] MarkupLMModel\n    - forward\n\n## MarkupLMForSequenceClassification\n\n[[autodoc]] MarkupLMForSequenceClassification\n    - forward\n\n## MarkupLMForTokenClassification\n\n[[autodoc]] MarkupLMForTokenClassification\n    - forward\n\n## MarkupLMForQuestionAnswering\n\n[[autodoc]] MarkupLMForQuestionAnswering\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5e3d6501-f584-4e2c-bb25-939f286264c0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5e3d6501-f584-4e2c-bb25-939f286264c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"06cb29f5-8f8f-425f-9962-0703275e6285\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2ce08cbe-3bdf-43ff-93a9-9e278075cef6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3e56931d1a11feb3c87f796a1f24bb508095200f765d983558270273643e5ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3067, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "doc_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "ref_doc_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "chroma:document": "# BEiT\n\n## Overview\n\nThe BEiT model was proposed in [BEiT: BERT Pre-Training of Image Transformers](https://arxiv.org/abs/2106.08254) by\nHangbo Bao, Li Dong and Furu Wei. Inspired by BERT, BEiT is the first paper that makes self-supervised pre-training of\nVision Transformers (ViTs) outperform supervised pre-training. Rather than pre-training the model to predict the class\nof an image (as done in the [original ViT paper](https://arxiv.org/abs/2010.11929)), BEiT models are pre-trained to\npredict visual tokens from the codebook of OpenAI's [DALL-E model](https://arxiv.org/abs/2102.12092) given masked\npatches.\n\nThe abstract from the paper is the following:\n\n*We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation\nfrom Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image\nmodeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image\npatches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first \"tokenize\" the original image into\nvisual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training\nobjective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we\ndirectly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder.\nExperimental results on image classification and semantic segmentation show that our model achieves competitive results\nwith previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K,\nsignificantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains\n86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The JAX/FLAX version of this model was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/beit).\n\n## Usage tips\n\n- BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They\n  outperform both the [original model (ViT)](vit) as well as [Data-efficient Image Transformers (DeiT)](deit) when fine-tuned on ImageNet-1K and CIFAR-100. You can check out demo notebooks regarding inference as well as\n  fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace\n  [`ViTFeatureExtractor`] by [`BeitImageProcessor`] and\n  [`ViTForImageClassification`] by [`BeitForImageClassification`]).\n- There's also a demo notebook available which showcases how to combine DALL-E's image tokenizer with BEiT for\n  performing masked image modeling. You can find it [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BEiT)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2ce08cbe-3bdf-43ff-93a9-9e278075cef6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2ce08cbe-3bdf-43ff-93a9-9e278075cef6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"06cb29f5-8f8f-425f-9962-0703275e6285\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5e3d6501-f584-4e2c-bb25-939f286264c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7c6257c64012a687ef28534ef3f5d7472681d437bd78e4e011ad0b1b84a67a7b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2a92012f-f585-44a5-8039-2ddd3e3bfa64\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3a81ece81201e5b44a5db6112158defb294e7caedc0eab21dfaeebe5df6d13cf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1640, \"end_char_idx\": 4555, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "doc_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "ref_doc_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "chroma:document": "For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K,\nsignificantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains\n86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The JAX/FLAX version of this model was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/beit).\n\n## Usage tips\n\n- BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They\n  outperform both the [original model (ViT)](vit) as well as [Data-efficient Image Transformers (DeiT)](deit) when fine-tuned on ImageNet-1K and CIFAR-100. You can check out demo notebooks regarding inference as well as\n  fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace\n  [`ViTFeatureExtractor`] by [`BeitImageProcessor`] and\n  [`ViTForImageClassification`] by [`BeitForImageClassification`]).\n- There's also a demo notebook available which showcases how to combine DALL-E's image tokenizer with BEiT for\n  performing masked image modeling. You can find it [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BEiT).\n- As the BEiT models expect each image to be of the same size (resolution), one can use\n  [`BeitImageProcessor`] to resize (or rescale) and normalize images for the model.\n- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of\n  each checkpoint. For example, `microsoft/beit-base-patch16-224` refers to a base-sized architecture with patch\n  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=microsoft/beit).\n- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of\n  14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the\n  relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position\n  bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to\n  pre-train a model from scratch, one needs to either set the `use_relative_position_bias` or the\n  `use_relative_position_bias` attribute of [`BeitConfig`] to `True` in order to add\n  position embeddings."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2a92012f-f585-44a5-8039-2ddd3e3bfa64
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2a92012f-f585-44a5-8039-2ddd3e3bfa64\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"06cb29f5-8f8f-425f-9962-0703275e6285\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2ce08cbe-3bdf-43ff-93a9-9e278075cef6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c6caa421c927a35787624f2129f6db7b32f04d2e0d8f9b7979b10d119556582f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1f9881b7-ce00-4b2d-9fa8-8ac3c55073e5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"383aea2d91e4685f986ab066d0b16b34fe375ff09ef83b09016af3dc98ca770f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3068, \"end_char_idx\": 5696, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "doc_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "ref_doc_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "chroma:document": "- As the BEiT models expect each image to be of the same size (resolution), one can use\n  [`BeitImageProcessor`] to resize (or rescale) and normalize images for the model.\n- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of\n  each checkpoint. For example, `microsoft/beit-base-patch16-224` refers to a base-sized architecture with patch\n  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=microsoft/beit).\n- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of\n  14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the\n  relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position\n  bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to\n  pre-train a model from scratch, one needs to either set the `use_relative_position_bias` or the\n  `use_relative_position_bias` attribute of [`BeitConfig`] to `True` in order to add\n  position embeddings.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/beit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> BEiT pre-training. Taken from the <a href=\"https://arxiv.org/abs/2106.08254\">original paper.</a> </small>\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BEiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n**Semantic segmentation**\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1f9881b7-ce00-4b2d-9fa8-8ac3c55073e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1f9881b7-ce00-4b2d-9fa8-8ac3c55073e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"06cb29f5-8f8f-425f-9962-0703275e6285\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2a92012f-f585-44a5-8039-2ddd3e3bfa64\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/beit\", \"title\": \"BEiT\", \"tokens\": 1832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a09b2578565fa327e02964582e2ba12d88ce22c94f5c7c22bf5f3268b0de0a47\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4004, \"end_char_idx\": 6752, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "doc_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "ref_doc_id": "06cb29f5-8f8f-425f-9962-0703275e6285", "chroma:document": "- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the\n  relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position\n  bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to\n  pre-train a model from scratch, one needs to either set the `use_relative_position_bias` or the\n  `use_relative_position_bias` attribute of [`BeitConfig`] to `True` in order to add\n  position embeddings.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/beit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> BEiT pre-training. Taken from the <a href=\"https://arxiv.org/abs/2106.08254\">original paper.</a> </small>\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BEiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n**Semantic segmentation**\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## BEiT specific outputs\n\n[[autodoc]] models.beit.modeling_beit.BeitModelOutputWithPooling\n\n[[autodoc]] models.beit.modeling_flax_beit.FlaxBeitModelOutputWithPooling\n\n## BeitConfig\n\n[[autodoc]] BeitConfig\n\n## BeitFeatureExtractor\n\n[[autodoc]] BeitFeatureExtractor\n    - __call__\n    - post_process_semantic_segmentation\n\n## BeitImageProcessor\n\n[[autodoc]] BeitImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n<frameworkcontent>\n<pt>\n\n## BeitModel\n\n[[autodoc]] BeitModel\n    - forward\n\n## BeitForMaskedImageModeling\n\n[[autodoc]] BeitForMaskedImageModeling\n    - forward\n\n## BeitForImageClassification\n\n[[autodoc]] BeitForImageClassification\n    - forward\n\n## BeitForSemanticSegmentation\n\n[[autodoc]] BeitForSemanticSegmentation\n    - forward\n\n</pt>\n<jax>\n\n## FlaxBeitModel\n\n[[autodoc]] FlaxBeitModel\n    - __call__\n\n## FlaxBeitForMaskedImageModeling\n\n[[autodoc]] FlaxBeitForMaskedImageModeling\n    - __call__\n\n## FlaxBeitForImageClassification\n\n[[autodoc]] FlaxBeitForImageClassification\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: dc27b9aa-1efa-4eb2-86db-119b0db536ee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roc_bert", "title": "RoCBert", "tokens": 843, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dc27b9aa-1efa-4eb2-86db-119b0db536ee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roc_bert\", \"title\": \"RoCBert\", \"tokens\": 843, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"432c2ba9-8033-4a19-b843-14f88d9b83d2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roc_bert\", \"title\": \"RoCBert\", \"tokens\": 843, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"29a2c7b768273c705aa7bbe6a595c6d18c24302b6a97371e897724b0777a6855\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2771, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "432c2ba9-8033-4a19-b843-14f88d9b83d2", "doc_id": "432c2ba9-8033-4a19-b843-14f88d9b83d2", "ref_doc_id": "432c2ba9-8033-4a19-b843-14f88d9b83d2", "chroma:document": "# RoCBert\n\n## Overview\n\nThe RoCBert model was proposed in [RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining](https://aclanthology.org/2022.acl-long.65.pdf)  by HuiSu, WeiweiShi, XiaoyuShen, XiaoZhou, TuoJi, JiaruiFang, JieZhou.\nIt's a pretrained Chinese language model that is robust under various forms of adversarial attacks.\n\nThe abstract from the paper is the following:\n\n*Large-scale pretrained language models have achieved SOTA results on NLP tasks. However, they have been shown\nvulnerable to adversarial attacks especially for logographic languages like Chinese. In this work, we propose\nROCBERT: a pretrained Chinese Bert that is robust to various forms of adversarial attacks like word perturbation,\nsynonyms, typos, etc. It is pretrained with the contrastive learning objective which maximizes the label consistency\nunder different synthesized adversarial examples. The model takes as input multimodal information including the\nsemantic, phonetic and visual features. We show all these features are important to the model robustness since the\nattack can be performed in all the three forms. Across 5 Chinese NLU tasks, ROCBERT outperforms strong baselines under\nthree blackbox adversarial algorithms without sacrificing the performance on clean testset. It also performs the best\nin the toxic content detection task under human-made attacks.*\n\nThis model was contributed by [weiweishi](https://huggingface.co/weiweishi).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## RoCBertConfig\n\n[[autodoc]] RoCBertConfig\n    - all\n\n## RoCBertTokenizer\n\n[[autodoc]] RoCBertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## RoCBertModel\n\n[[autodoc]] RoCBertModel\n    - forward\n\n## RoCBertForPreTraining\n\n[[autodoc]] RoCBertForPreTraining\n    - forward\n\n## RoCBertForCausalLM\n\n[[autodoc]] RoCBertForCausalLM\n    - forward\n\n## RoCBertForMaskedLM\n\n[[autodoc]] RoCBertForMaskedLM\n    - forward\n\n## RoCBertForSequenceClassification\n\n[[autodoc]] transformers.RoCBertForSequenceClassification\n    - forward\n\n## RoCBertForMultipleChoice\n\n[[autodoc]] transformers.RoCBertForMultipleChoice\n    - forward\n\n## RoCBertForTokenClassification\n\n[[autodoc]] transformers.RoCBertForTokenClassification\n    - forward\n\n## RoCBertForQuestionAnswering\n\n[[autodoc]] RoCBertForQuestionAnswering\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4dd6b10c-54d0-4c1d-9002-e3fdf5aca6f7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swiftformer", "title": "SwiftFormer", "tokens": 774, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4dd6b10c-54d0-4c1d-9002-e3fdf5aca6f7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swiftformer\", \"title\": \"SwiftFormer\", \"tokens\": 774, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b988ca8a-9196-400f-a20b-e90255a7d0dd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swiftformer\", \"title\": \"SwiftFormer\", \"tokens\": 774, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a5d9bc408c2943dc09dc397aa8c00bd3a5e2a5fb6e387b674d59f1a10a4a0a40\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2865, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b988ca8a-9196-400f-a20b-e90255a7d0dd", "doc_id": "b988ca8a-9196-400f-a20b-e90255a7d0dd", "ref_doc_id": "b988ca8a-9196-400f-a20b-e90255a7d0dd", "chroma:document": "# SwiftFormer\n\n## Overview\n\nThe SwiftFormer model was proposed in [SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications](https://arxiv.org/abs/2303.15446) by Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan.\n\nThe SwiftFormer paper introduces a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations in the self-attention computation with linear element-wise multiplications. A series of models called 'SwiftFormer' is built based on this, which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Even their small variant achieves 78.5% top-1 ImageNet1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2\u00d7 faster compared to MobileViT-v2.\n\nThe abstract from the paper is the following:\n\n*Self-attention has become a defacto choice for capturing global context in various vision applications. However, its quadratic computational complexity with respect to image resolution limits its use in real-time applications, especially for deployment on resource-constrained mobile devices. Although hybrid approaches have been proposed to combine the advantages of convolutions and self-attention for a better speed-accuracy trade-off, the expensive matrix multiplication operations in self-attention remain a bottleneck. In this work, we introduce a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations with linear element-wise multiplications. Our design shows that the key-value interaction can be replaced with a linear layer without sacrificing any accuracy. Unlike previous state-of-the-art methods, our efficient formulation of self-attention enables its usage at all stages of the network. Using our proposed efficient additive attention, we build a series of models called \"SwiftFormer\" which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Our small variant achieves 78.5% top-1 ImageNet-1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2x faster compared to MobileViT-v2.*\n\nThis model was contributed by [shehan97](https://huggingface.co/shehan97). The TensorFlow version was contributed by [joaocmd](https://huggingface.co/joaocmd).\nThe original code can be found [here](https://github.com/Amshaker/SwiftFormer).\n\n## SwiftFormerConfig\n\n[[autodoc]] SwiftFormerConfig\n\n## SwiftFormerModel\n\n[[autodoc]] SwiftFormerModel\n    - forward\n\n## SwiftFormerForImageClassification\n\n[[autodoc]] SwiftFormerForImageClassification\n    - forward\n\n## TFSwiftFormerModel\n\n[[autodoc]] TFSwiftFormerModel\n    - call\n\n## TFSwiftFormerForImageClassification\n\n[[autodoc]] TFSwiftFormerForImageClassification\n    - call"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ef495369-8595-40bd-8491-5c8f2cfd2e8a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ef495369-8595-40bd-8491-5c8f2cfd2e8a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2d2f92f2-3109-4298-9775-8aa63651dcdb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"805db382-3599-403e-9150-e7f4a675258f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d9383a3826a20c50a2c23a9d66c5773e749623bac8849a6c1402d3447bbd68c1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3651, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "ref_doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "chroma:document": "# SeamlessM4T-v2\n\n## Overview\n\nThe SeamlessM4T-v2 model was proposed in [Seamless: Multilingual Expressive and Streaming Speech Translation](https://ai.meta.com/research/publications/seamless-multilingual-expressive-and-streaming-speech-translation/) by the Seamless Communication team from Meta AI.\n\nSeamlessM4T-v2 is a collection of models designed to provide high quality translation, allowing people from different linguistic communities to communicate effortlessly through speech and text. It is an improvement on the [previous version](https://huggingface.co/docs/transformers/main/model_doc/seamless_m4t). For more details on the differences between v1 and v2, refer to section [Difference with SeamlessM4T-v1](#difference-with-seamlessm4t-v1).\n\nSeamlessM4T-v2 enables multiple tasks without relying on separate models:\n\n- Speech-to-speech translation (S2ST)\n- Speech-to-text translation (S2TT)\n- Text-to-speech translation (T2ST)\n- Text-to-text translation (T2TT)\n- Automatic speech recognition (ASR)\n\n[`SeamlessM4Tv2Model`] can perform all the above tasks, but each task also has its own dedicated sub-model.\n\nThe abstract from the paper is the following:\n\n*Recent advancements in automatic speech translation have dramatically expanded language coverage, improved multimodal capabilities, and enabled a wide range of tasks and functionalities. That said, large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model\u2014SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one\u2019s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 805db382-3599-403e-9150-e7f4a675258f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"805db382-3599-403e-9150-e7f4a675258f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2d2f92f2-3109-4298-9775-8aa63651dcdb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ef495369-8595-40bd-8491-5c8f2cfd2e8a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"539d35860868f918b8be0b56a5e51753bea3cff3bac26fb4538bf7585a567744\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e9a3b04b-d8a5-45cc-ab72-7a8b60a4a632\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"325a8ef556a08a12499522197b9defa003d2d2842b57cbc64fa2af0d72006f30\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1669, \"end_char_idx\": 5084, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "ref_doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "chroma:document": "First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model\u2014SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one\u2019s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work\u2014including models, code, and a watermark detector\u2014are publicly released and accessible at the link below.*\n\n## Usage\n\nIn the following example, we'll load an Arabic audio sample and an English text sample and convert them into Russian speech and French text.\n\nFirst, load the processor and a checkpoint of the model:\n\n```python\n>>> from transformers import AutoProcessor, SeamlessM4Tv2Model\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n>>> model = SeamlessM4Tv2Model.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nYou can seamlessly use this model on text or on audio, to generated either translated text or translated audio.\n\nHere is how to use the processor to process text and audio:\n\n```python\n>>> # let's load an audio sample from an Arabic speech corpus\n>>> from datasets import load_dataset\n>>> dataset = load_dataset(\"arabic_speech_corpus\", split=\"test\", streaming=True)\n>>> audio_sample = next(iter(dataset))[\"audio\"]\n\n>>> # now, process it\n>>> audio_inputs = processor(audios=audio_sample[\"array\"], return_tensors=\"pt\")\n\n>>> # now, process some English text as well\n>>> text_inputs = processor(text = \"Hello, my dog is cute\", src_lang=\"eng\", return_tensors=\"pt\")\n```"}


