# embeddings_queue

created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ac2ad1d5-79a9-4d2c-9cfb-9896d4a5a47c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/efficientnet/", "title": "EfficientNet", "tokens": 631, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ac2ad1d5-79a9-4d2c-9cfb-9896d4a5a47c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientnet/\", \"title\": \"EfficientNet\", \"tokens\": 631, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dc73b96c-081e-431b-82c3-dbf505d1458f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientnet/\", \"title\": \"EfficientNet\", \"tokens\": 631, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"417c8b905794a2c74cd6706e0ac2e2e3fc4cc370ba32abda3a2d3a5851984d9c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2129, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dc73b96c-081e-431b-82c3-dbf505d1458f", "doc_id": "dc73b96c-081e-431b-82c3-dbf505d1458f", "ref_doc_id": "dc73b96c-081e-431b-82c3-dbf505d1458f", "chroma:document": "# EfficientNet\n\n## Overview\n\nThe EfficientNet model was proposed in [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://arxiv.org/abs/1905.11946) \nby Mingxing Tan and Quoc V. Le. EfficientNets are a family of image classification models, which achieve state-of-the-art accuracy, yet being an order-of-magnitude smaller and faster than previous models.\n\nThe abstract from the paper is the following:\n\n*Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.\nTo go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters.*\n\nThis model was contributed by [adirik](https://huggingface.co/adirik).\nThe original code can be found [here](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet).\n\n\n## EfficientNetConfig\n\n[[autodoc]] EfficientNetConfig\n\n## EfficientNetImageProcessor\n\n[[autodoc]] EfficientNetImageProcessor\n    - preprocess\n\n## EfficientNetModel\n\n[[autodoc]] EfficientNetModel\n    - forward\n\n## EfficientNetForImageClassification\n\n[[autodoc]] EfficientNetForImageClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1ff0be3f-dc69-4840-9a72-55b9ad1bccdd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flan-ul2/", "title": "FLAN-UL2", "tokens": 675, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1ff0be3f-dc69-4840-9a72-55b9ad1bccdd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flan-ul2/\", \"title\": \"FLAN-UL2\", \"tokens\": 675, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0a0af45d-f530-4c20-832b-a0634e1f38c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flan-ul2/\", \"title\": \"FLAN-UL2\", \"tokens\": 675, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fffc17d0f56618bd7508c7b334b7739dbbbb84ae1f09d1f0027a5143d66c0619\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2170, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0a0af45d-f530-4c20-832b-a0634e1f38c0", "doc_id": "0a0af45d-f530-4c20-832b-a0634e1f38c0", "ref_doc_id": "0a0af45d-f530-4c20-832b-a0634e1f38c0", "chroma:document": "# FLAN-UL2\n\n## Overview\n\nFlan-UL2 is an encoder decoder model based on the T5 architecture. It uses the same configuration as the [UL2](ul2) model released earlier last year. \nIt was fine tuned using the \"Flan\" prompt tuning and dataset collection. Similar to `Flan-T5`,  one can directly use FLAN-UL2 weights without finetuning the model:\n\nAccording to the original blog here are the notable improvements:\n\n- The original UL2 model was only trained with receptive field of 512, which made it non-ideal for N-shot prompting where N is large.\n- The Flan-UL2 checkpoint uses a receptive field of 2048 which makes it more usable for few-shot in-context learning.\n- The original UL2 model also had mode switch tokens that was rather mandatory to get good performance. However, they were a little cumbersome as this requires often some changes during inference or finetuning. In this update/change, we continue training UL2 20B for an additional 100k steps (with small batch) to forget \u201cmode tokens\u201d before applying Flan instruction tuning. This Flan-UL2 checkpoint does not require mode tokens anymore.\nGoogle has released the following variants:\n\nThe original checkpoints can be found [here](https://github.com/google-research/t5x/blob/main/docs/models.md#flan-ul2-checkpoints).\n\n\n## Running on low resource devices\n\nThe model is pretty heavy (~40GB in half precision) so if you just want to run the model, make sure you load your model in 8bit, and use `device_map=\"auto\"` to make sure  you don't have any OOM issue!\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/flan-ul2\", load_in_8bit=True, device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/flan-ul2\")\n\n>>> inputs = tokenizer(\"A step by step recipe to make bolognese pasta:\", return_tensors=\"pt\")\n>>> outputs = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['In a large skillet, brown the ground beef and onion over medium heat. Add the garlic']\n```\n\n<Tip>\n\nRefer to [T5's documentation page](t5) for API reference, tips, code examples and notebooks. \n\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8cb26323-431a-44fa-a7cc-1007792711b1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nougat/", "title": "Nougat", "tokens": 1239, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8cb26323-431a-44fa-a7cc-1007792711b1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat/\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab8e5a9f-9684-445c-99bc-1fe10623823b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat/\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b81dfe2c636ec68115b615126a5286f6977192f03528e1ece4187e5cdcafc95d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f51b0e60-01ae-4627-9a31-e0625acd21f2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"566a7aba6b86094d47703f50a0e286c15a07946e9ed9751428c39be649d81395\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2716, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab8e5a9f-9684-445c-99bc-1fe10623823b", "doc_id": "ab8e5a9f-9684-445c-99bc-1fe10623823b", "ref_doc_id": "ab8e5a9f-9684-445c-99bc-1fe10623823b", "chroma:document": "# Nougat\n\n## Overview\n\nThe Nougat model was proposed in [Nougat: Neural Optical Understanding for Academic Documents](https://arxiv.org/abs/2308.13418) by\nLukas Blecher, Guillem Cucurull, Thomas Scialom, Robert Stojnic. Nougat uses the same architecture as [Donut](donut), meaning an image Transformer\nencoder and an autoregressive text Transformer decoder to translate scientific PDFs to markdown, enabling easier access to them.\n\nThe abstract from the paper is the following:\n\n*Scientific knowledge is predominantly stored in books and scientific journals, often in the form of PDFs. However, the PDF format leads to a loss of semantic information, particularly for mathematical expressions. We propose Nougat (Neural Optical Understanding for Academic Documents), a Visual Transformer model that performs an Optical Character Recognition (OCR) task for processing scientific documents into a markup language, and demonstrate the effectiveness of our model on a new dataset of scientific documents. The proposed approach offers a promising solution to enhance the accessibility of scientific knowledge in the digital age, by bridging the gap between human-readable documents and machine-readable text. We release the models and code to accelerate future work on scientific text recognition.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/nougat_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Nougat high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2308.13418\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/facebookresearch/nougat).\n\n## Usage tips\n\n- The quickest way to get started with Nougat is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Nougat), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- Nougat is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework. The model is identical to [Donut](donut) in terms of architecture.\n\n## Inference\n\nNougat's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`NougatImageProcessor`] class is responsible for preprocessing the input image and\n[`NougatTokenizerFast`] decodes the generated target tokens to the target string. The\n[`NougatProcessor`] wraps [`NougatImageProcessor`] and [`NougatTokenizerFast`] classes\ninto a single instance to both extract the input features and decode the predicted token ids."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f51b0e60-01ae-4627-9a31-e0625acd21f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nougat/", "title": "Nougat", "tokens": 1239, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f51b0e60-01ae-4627-9a31-e0625acd21f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat/\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab8e5a9f-9684-445c-99bc-1fe10623823b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat/\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b81dfe2c636ec68115b615126a5286f6977192f03528e1ece4187e5cdcafc95d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8cb26323-431a-44fa-a7cc-1007792711b1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat/\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a38354446e3479c8d8d8726a285a107fff80709bc07f7e929e6400be40eb4bd8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b9028f8f-60d9-4968-bfbc-6d0b1f98c858\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ff184a9927f33c2670b373752f1b149dbeed551acb98ac6f8a839f46df964899\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1204, \"end_char_idx\": 4342, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab8e5a9f-9684-445c-99bc-1fe10623823b", "doc_id": "ab8e5a9f-9684-445c-99bc-1fe10623823b", "ref_doc_id": "ab8e5a9f-9684-445c-99bc-1fe10623823b", "chroma:document": "We release the models and code to accelerate future work on scientific text recognition.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/nougat_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Nougat high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2308.13418\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/facebookresearch/nougat).\n\n## Usage tips\n\n- The quickest way to get started with Nougat is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Nougat), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- Nougat is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework. The model is identical to [Donut](donut) in terms of architecture.\n\n## Inference\n\nNougat's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`NougatImageProcessor`] class is responsible for preprocessing the input image and\n[`NougatTokenizerFast`] decodes the generated target tokens to the target string. The\n[`NougatProcessor`] wraps [`NougatImageProcessor`] and [`NougatTokenizerFast`] classes\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step PDF transcription\n\n```py\n>>> from huggingface_hub import hf_hub_download\n>>> import re\n>>> from PIL import Image\n\n>>> from transformers import NougatProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = NougatProcessor.from_pretrained(\"facebook/nougat-base\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"facebook/nougat-base\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # prepare PDF image for the model\n>>> filepath = hf_hub_download(repo_id=\"hf-internal-testing/fixtures_docvqa\", filename=\"nougat_paper.png\", repo_type=\"dataset\")\n>>> image = Image.open(filepath)\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> # generate transcription (here we only generate 30 tokens)\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     min_length=1,\n...     max_new_tokens=30,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n... )\n\n>>> sequence = processor.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> sequence = processor.post_process_generation(sequence, fix_markdown=False)\n>>> # note: we're using repr here such for the sake of printing the \\n characters, feel free to just print the sequence\n>>> print(repr(sequence))\n'\\n\\n# Nougat: Neural Optical Understanding for Academic Documents\\n\\n Lukas Blecher\\n\\nCorrespondence to: lblecher@'\n```\n\nSee the [model hub](https://huggingface.co/models?filter=nougat) to look for Nougat checkpoints.\n\n<Tip>\n\nThe model is identical to [Donut](donut) in terms of architecture."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b9028f8f-60d9-4968-bfbc-6d0b1f98c858
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nougat/", "title": "Nougat", "tokens": 1239, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b9028f8f-60d9-4968-bfbc-6d0b1f98c858\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat/\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab8e5a9f-9684-445c-99bc-1fe10623823b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat/\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b81dfe2c636ec68115b615126a5286f6977192f03528e1ece4187e5cdcafc95d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f51b0e60-01ae-4627-9a31-e0625acd21f2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nougat/\", \"title\": \"Nougat\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ed311caed5b09bb37e87b1c9079efc48503abdd21de306926993077ad228edf6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2718, \"end_char_idx\": 4653, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab8e5a9f-9684-445c-99bc-1fe10623823b", "doc_id": "ab8e5a9f-9684-445c-99bc-1fe10623823b", "ref_doc_id": "ab8e5a9f-9684-445c-99bc-1fe10623823b", "chroma:document": "- Step-by-step PDF transcription\n\n```py\n>>> from huggingface_hub import hf_hub_download\n>>> import re\n>>> from PIL import Image\n\n>>> from transformers import NougatProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = NougatProcessor.from_pretrained(\"facebook/nougat-base\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"facebook/nougat-base\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # prepare PDF image for the model\n>>> filepath = hf_hub_download(repo_id=\"hf-internal-testing/fixtures_docvqa\", filename=\"nougat_paper.png\", repo_type=\"dataset\")\n>>> image = Image.open(filepath)\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> # generate transcription (here we only generate 30 tokens)\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     min_length=1,\n...     max_new_tokens=30,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n... )\n\n>>> sequence = processor.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> sequence = processor.post_process_generation(sequence, fix_markdown=False)\n>>> # note: we're using repr here such for the sake of printing the \\n characters, feel free to just print the sequence\n>>> print(repr(sequence))\n'\\n\\n# Nougat: Neural Optical Understanding for Academic Documents\\n\\n Lukas Blecher\\n\\nCorrespondence to: lblecher@'\n```\n\nSee the [model hub](https://huggingface.co/models?filter=nougat) to look for Nougat checkpoints.\n\n<Tip>\n\nThe model is identical to [Donut](donut) in terms of architecture.\n\n</Tip>\n\n## NougatImageProcessor\n\n[[autodoc]] NougatImageProcessor\n    - preprocess\n\n## NougatTokenizerFast\n\n[[autodoc]] NougatTokenizerFast\n\n## NougatProcessor\n\n[[autodoc]] NougatProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n    - post_process_generation"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cccfa0af-6320-46cc-a5e9-154600069b0c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava/", "title": "LLaVa", "tokens": 1603, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cccfa0af-6320-46cc-a5e9-154600069b0c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"403ec9a8-4a78-45a3-aaf3-d60dfe867088\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2fdfb8abda70a7f5bf575c859b12cbe80a39ab6652e8f6fec7594b2d767aa532\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8b860164-a849-4c23-918b-3da878e09d7b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7f86e95301b0fa84e31546635c55a3d2098008eae10dedaf1e0d6c2a53263153\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3120, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "doc_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "ref_doc_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "chroma:document": "# LLaVa\n\n## Overview\n\nLLaVa is an open-source chatbot trained by fine-tuning LlamA/Vicuna on GPT-generated multimodal instruction-following data. It is an auto-regressive language model, based on the transformer architecture. In other words, it is an multi-modal version of LLMs fine-tuned for chat / instructions.\n\nThe LLaVa model was proposed in [Visual Instruction Tuning](https://arxiv.org/abs/2304.08485) and improved in [Improved Baselines with Visual Instruction Tuning](https://arxiv.org/pdf/2310.03744) by Haotian Liu, Chunyuan Li, Yuheng Li and Yong Jae Lee.\n\nThe abstract from the paper is the following:\n\n*Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in \u223c1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/llava_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> LLaVa architecture. Taken from the <a href=\"https://arxiv.org/abs/2304.08485\">original paper.</a> </small>\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ) and [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/main/llava).\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities, as follows:\n\n```python\nfrom transformers import AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/llava-1.5-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8b860164-a849-4c23-918b-3da878e09d7b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava/", "title": "LLaVa", "tokens": 1603, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8b860164-a849-4c23-918b-3da878e09d7b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"403ec9a8-4a78-45a3-aaf3-d60dfe867088\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2fdfb8abda70a7f5bf575c859b12cbe80a39ab6652e8f6fec7594b2d767aa532\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cccfa0af-6320-46cc-a5e9-154600069b0c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"775a23ba1a1241474b9e467086bfbf96c4188e4e0de5ee4cb3b902a3c23c597a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"43a9dc3d-8ce8-4fa8-aa1c-d6d3c4365519\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ddfdd40c163088f405f669c5eab9c957d2a5fb44425db85c0b2b0e7e5dbf4520\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1552, \"end_char_idx\": 3649, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "doc_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "ref_doc_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2304.08485\">original paper.</a> </small>\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ) and [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/main/llava).\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities, as follows:\n\n```python\nfrom transformers import AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/llava-1.5-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"USER: <image>\\n<What\u2019s shown in this image? ASSISTANT: This image shows a red stop sign.</s>USER: Describe the image in more details."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 43a9dc3d-8ce8-4fa8-aa1c-d6d3c4365519
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava/", "title": "LLaVa", "tokens": 1603, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"43a9dc3d-8ce8-4fa8-aa1c-d6d3c4365519\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"403ec9a8-4a78-45a3-aaf3-d60dfe867088\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2fdfb8abda70a7f5bf575c859b12cbe80a39ab6652e8f6fec7594b2d767aa532\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8b860164-a849-4c23-918b-3da878e09d7b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"441ae34c64af355a05fe6aed1c32463d0b278b770aae3f3ce7c9c6bc8682e675\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c861bad3-b63a-403b-b8e7-81d7d7c6459e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"029d74141add494a0cc659b8bb676883fb4c9cd5053a1f2c0f8945cd8eb4b9d4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1989, \"end_char_idx\": 4578, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "doc_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "ref_doc_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "chroma:document": "Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities, as follows:\n\n```python\nfrom transformers import AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/llava-1.5-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"USER: <image>\\n<What\u2019s shown in this image? ASSISTANT: This image shows a red stop sign.</s>USER: Describe the image in more details. ASSISTANT:\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of prompt formats accepted by each llava checkpoint:\n\n[llava-interleave models](https://huggingface.co/collections/llava-hf/llava-interleave-668e19a97da0036aad4a2f19) requires the following format:\n```bash\n\"<|im_start|>user <image>\\nWhat is shown in this image?<|im_end|><|im_start|>assistant\"\n```\n\nFor multiple turns conversation:\n\n```bash\n\"<|im_start|>user <image>\\n<prompt1><|im_end|><|im_start|>assistant <answer1><|im_end|><|im_start|>user <image>\\n<prompt1><|im_end|><|im_start|>assistant \"\n```\n\n[llava-1.5 models](https://huggingface.co/collections/llava-hf/llava-15-65f762d5b6941db5c2ba07e0) requires the following format:\n```bash\n\"USER: <image>\\n<prompt> ASSISTANT:\"\n```\n\nFor multiple turns conversation:\n\n```bash\n\"USER: <image>\\n<prompt1> ASSISTANT: <answer1></s>USER: <prompt2> ASSISTANT: <answer2></s>USER: <prompt3> ASSISTANT:\"\n```"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c861bad3-b63a-403b-b8e7-81d7d7c6459e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava/", "title": "LLaVa", "tokens": 1603, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c861bad3-b63a-403b-b8e7-81d7d7c6459e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"403ec9a8-4a78-45a3-aaf3-d60dfe867088\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2fdfb8abda70a7f5bf575c859b12cbe80a39ab6652e8f6fec7594b2d767aa532\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"43a9dc3d-8ce8-4fa8-aa1c-d6d3c4365519\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava/\", \"title\": \"LLaVa\", \"tokens\": 1603, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"319a86d9d81cd0f4abcc3c5ffb991d8aca59684de93376c8b7b330c2a3fc2c21\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3257, \"end_char_idx\": 5545, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "doc_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "ref_doc_id": "403ec9a8-4a78-45a3-aaf3-d60dfe867088", "chroma:document": "\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"USER: <image>\\n<What\u2019s shown in this image? ASSISTANT: This image shows a red stop sign.</s>USER: Describe the image in more details. ASSISTANT:\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of prompt formats accepted by each llava checkpoint:\n\n[llava-interleave models](https://huggingface.co/collections/llava-hf/llava-interleave-668e19a97da0036aad4a2f19) requires the following format:\n```bash\n\"<|im_start|>user <image>\\nWhat is shown in this image?<|im_end|><|im_start|>assistant\"\n```\n\nFor multiple turns conversation:\n\n```bash\n\"<|im_start|>user <image>\\n<prompt1><|im_end|><|im_start|>assistant <answer1><|im_end|><|im_start|>user <image>\\n<prompt1><|im_end|><|im_start|>assistant \"\n```\n\n[llava-1.5 models](https://huggingface.co/collections/llava-hf/llava-15-65f762d5b6941db5c2ba07e0) requires the following format:\n```bash\n\"USER: <image>\\n<prompt> ASSISTANT:\"\n```\n\nFor multiple turns conversation:\n\n```bash\n\"USER: <image>\\n<prompt1> ASSISTANT: <answer1></s>USER: <prompt2> ASSISTANT: <answer2></s>USER: <prompt3> ASSISTANT:\"\n```\n\n\n### Using Flash Attention 2\n\nFlash Attention 2 is an even faster, optimized version of the previous optimization, please refer to the [Flash Attention 2 section of performance docs](https://huggingface.co/docs/transformers/perf_infer_gpu_one).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BEiT.\n\n<PipelineTag pipeline=\"image-to-text\"/>\n\n- A [Google Colab demo](https://colab.research.google.com/drive/1qsl6cd2c8gGtEW1xV5io7S8NHh-Cp1TV?usp=sharing) on how to run Llava on a free-tier Google colab instance leveraging 4-bit inference.\n- A [similar notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LLaVa/Inference_with_LLaVa_for_multimodal_generation.ipynb) showcasing batched inference. \ud83c\udf0e\n\n\n## LlavaConfig\n\n[[autodoc]] LlavaConfig\n\n## LlavaProcessor\n\n[[autodoc]] LlavaProcessor\n\n## LlavaForConditionalGeneration\n\n[[autodoc]] LlavaForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e1272782-f69f-46f1-b7b4-bd7c3a10ea32
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron_gpt2/", "title": "MegatronGPT2", "tokens": 1079, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e1272782-f69f-46f1-b7b4-bd7c3a10ea32\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2/\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"715e3f69-37e0-4b25-bd37-3be1ee614481\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2/\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2483df587dbf9176e4bd3941b9b63856020e50ec9bca8123e97f561922e443a7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"adcdd5be-eb5f-413c-b03b-f7badbacb623\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7583c9d3385b51d21aee21b4d8d703f3579209059f1a108a558d7b60900fec71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2851, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "715e3f69-37e0-4b25-bd37-3be1ee614481", "doc_id": "715e3f69-37e0-4b25-bd37-3be1ee614481", "ref_doc_id": "715e3f69-37e0-4b25-bd37-3be1ee614481", "chroma:document": "# MegatronGPT2\n\n## Overview\n\nThe MegatronGPT2 model was proposed in [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model\nParallelism](https://arxiv.org/abs/1909.08053) by Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley,\nJared Casper and Bryan Catanzaro.\n\nThe abstract from the paper is the following:\n\n*Recent work in language modeling demonstrates that training large transformer models advances the state of the art in\nNatural Language Processing applications. However, very large models can be quite difficult to train due to memory\nconstraints. In this work, we present our techniques for training very large transformer models and implement a simple,\nefficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our\napproach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model\nparallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We\nillustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain\n15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline\nthat sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance\nthe state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9\nbillion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in\nBERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we\nachieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA\naccuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy\nof 89.4%).*\n\nThis model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, it \ncontains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [GPT2-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_lm_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: adcdd5be-eb5f-413c-b03b-f7badbacb623
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron_gpt2/", "title": "MegatronGPT2", "tokens": 1079, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"adcdd5be-eb5f-413c-b03b-f7badbacb623\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2/\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"715e3f69-37e0-4b25-bd37-3be1ee614481\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2/\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2483df587dbf9176e4bd3941b9b63856020e50ec9bca8123e97f561922e443a7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e1272782-f69f-46f1-b7b4-bd7c3a10ea32\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron_gpt2/\", \"title\": \"MegatronGPT2\", \"tokens\": 1079, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f9860099b83d9f667e9c7fc0ed3b5ecbfe9df677549753a1cb6710ebc6742d52\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1337, \"end_char_idx\": 3743, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "715e3f69-37e0-4b25-bd37-3be1ee614481", "doc_id": "715e3f69-37e0-4b25-bd37-3be1ee614481", "ref_doc_id": "715e3f69-37e0-4b25-bd37-3be1ee614481", "chroma:document": "To demonstrate that large language models can further advance\nthe state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9\nbillion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in\nBERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we\nachieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA\naccuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy\nof 89.4%).*\n\nThis model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, it \ncontains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [GPT2-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_lm_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).\n\nAlternatively, you can directly download the checkpoints using:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_lm_345m/versions/v0.0/zip -O\nmegatron_gpt2_345m_v0_0.zip\n```\n\nOnce you have obtained the checkpoint from NVIDIA GPU Cloud (NGC), you have to convert it to a format that will easily\nbe loaded by Hugging Face Transformers GPT2 implementation.\n\nThe following command allows you to do the conversion. We assume that the folder `models/megatron_gpt2` contains\n`megatron_gpt2_345m_v0_0.zip` and that the command is run from that folder:\n\n```bash\npython3 $PATH_TO_TRANSFORMERS/models/megatron_gpt2/convert_megatron_gpt2_checkpoint.py megatron_gpt2_345m_v0_0.zip\n```\n\n<Tip> \n\n MegatronGPT2 architecture is the same as OpenAI GPT-2 . Refer to [GPT-2 documentation](gpt2) for information on \n configuration classes and their parameters.  \n\n </Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 09bb7859-a22e-4b98-b97e-fa58ada36f2f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/opt/", "title": "OPT", "tokens": 2021, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"09bb7859-a22e-4b98-b97e-fa58ada36f2f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b80bc382-38ad-40d4-a51c-350c59903a91\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5ba997ba45d67f36caf2611276438ec7f691f96ca5c0b3f2607b315847813ceb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f57fbec2-2176-4dbd-b055-0ad52414dadf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e4444a389432fde452460c98f8865b9f33ca21cbe71bd577e888a94db9930270\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3068, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "doc_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "ref_doc_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "chroma:document": "# OPT\n\n## Overview\n\nThe OPT model was proposed in [Open Pre-trained Transformer Language Models](https://arxiv.org/pdf/2205.01068) by Meta AI.\nOPT is a series of open-sourced large causal language models which perform similar in performance to GPT3.\n\nThe abstract from the paper is the following:\n\n*Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.*\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ), [Younes Belkada](https://huggingface.co/ybelkada), and [Patrick Von Platen](https://huggingface.co/patrickvonplaten).\nThe original code can be found [here](https://github.com/facebookresearch/metaseq).\n\nTips:\n- OPT has the same architecture as [`BartDecoder`].\n- Contrary to GPT2, OPT adds the EOS token `</s>` to the beginning of every prompt.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with OPT. If you're\ninterested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\" />\n\n- A notebook on [fine-tuning OPT with PEFT, bitsandbytes, and Transformers](https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing). \ud83c\udf0e\n- A blog post on [decoding strategies with OPT](https://huggingface.co/blog/introducing-csearch#62-example-two---opt).\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`OPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f57fbec2-2176-4dbd-b055-0ad52414dadf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/opt/", "title": "OPT", "tokens": 2021, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f57fbec2-2176-4dbd-b055-0ad52414dadf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b80bc382-38ad-40d4-a51c-350c59903a91\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5ba997ba45d67f36caf2611276438ec7f691f96ca5c0b3f2607b315847813ceb\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"09bb7859-a22e-4b98-b97e-fa58ada36f2f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"99b808808bf26144feb140d9ab51fe841494333ab530c425d6ef31edfd496aba\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2c492f85-691a-4c09-9fea-6f487700d285\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a6fa7f1420d2454099f517097181f8a15a52c5e9a7c24213f26a4bb980cf781\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1575, \"end_char_idx\": 4287, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "doc_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "ref_doc_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "chroma:document": "## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with OPT. If you're\ninterested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\" />\n\n- A notebook on [fine-tuning OPT with PEFT, bitsandbytes, and Transformers](https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing). \ud83c\udf0e\n- A blog post on [decoding strategies with OPT](https://huggingface.co/blog/introducing-csearch#62-example-two---opt).\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`OPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling).\n\n<PipelineTag pipeline=\"text-classification\" />\n\n- [Text classification task guide](sequence_classification.md)\n- [`OPTForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n\n<PipelineTag pipeline=\"question-answering\" />\n\n- [`OPTForQuestionAnswering`] is supported by this [question answering example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter\n  of the \ud83e\udd17 Hugging Face Course.\n\n\u26a1\ufe0f Inference\n\n- A blog post on [How \ud83e\udd17 Accelerate runs very large models thanks to PyTorch](https://huggingface.co/blog/accelerate-large-models) with OPT."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2c492f85-691a-4c09-9fea-6f487700d285
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/opt/", "title": "OPT", "tokens": 2021, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2c492f85-691a-4c09-9fea-6f487700d285\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b80bc382-38ad-40d4-a51c-350c59903a91\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5ba997ba45d67f36caf2611276438ec7f691f96ca5c0b3f2607b315847813ceb\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f57fbec2-2176-4dbd-b055-0ad52414dadf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5df4a8532cb8d40d7558b55e63f823ff4962d855600be732734f35070a0456d2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9070f903-04f8-4c2b-b25f-604c71ba9028\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59cd8cf2338cf35f73d8c4972a32f31ed246242063a4619c0e1b9bb0d1d8283a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2754, \"end_char_idx\": 6804, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "doc_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "ref_doc_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "chroma:document": "- [`TFOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxOPTForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling).\n\n<PipelineTag pipeline=\"text-classification\" />\n\n- [Text classification task guide](sequence_classification.md)\n- [`OPTForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n\n<PipelineTag pipeline=\"question-answering\" />\n\n- [`OPTForQuestionAnswering`] is supported by this [question answering example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter\n  of the \ud83e\udd17 Hugging Face Course.\n\n\u26a1\ufe0f Inference\n\n- A blog post on [How \ud83e\udd17 Accelerate runs very large models thanks to PyTorch](https://huggingface.co/blog/accelerate-large-models) with OPT.\n\n\n## Combining OPT and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import OPTForCausalLM, GPT2Tokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = OPTForCausalLM.from_pretrained(\"facebook/opt-350m\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = GPT2Tokenizer.from_pretrained(\"facebook/opt-350m\")\n\n>>> prompt = (\"A chat between a curious human and the Statue of Liberty.\\n\\nHuman: What is your name?\\nStatue: I am the \"\n              \"Statue of Liberty.\\nHuman: Where do you live?\\nStatue: New York City.\\nHuman: How long have you lived \"\n              \"there?\")\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=30, do_sample=False)\n>>> tokenizer.batch_decode(generated_ids)[0]\n'</s>A chat between a curious human and the Statue of Liberty.\\n\\nHuman: What is your name?\\nStatue: I am the Statue of Liberty.\\nHuman: Where do you live?\\nStatue: New York City.\\nHuman: How long have you lived there?\\nStatue: I have lived here for about a year.\\nHuman: What is your favorite place to eat?\\nStatue: I love'\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `facebook/opt-2.7b` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.\n\n<div style=\"text-align: center\">\n<img src=\"https://user-images.githubusercontent.com/49240599/281101546-d2fca6d2-ee44-48f3-9534-ba8d5bee4531.png\">\n</div>\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `facebook/opt-350m` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.\n\n<div style=\"text-align: center\">\n<img src=\"https://user-images.githubusercontent.com/49240599/281101682-d1144e90-0dbc-46f4-8fc8-c6206cb793c9.png\">\n</div>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9070f903-04f8-4c2b-b25f-604c71ba9028
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/opt/", "title": "OPT", "tokens": 2021, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9070f903-04f8-4c2b-b25f-604c71ba9028\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b80bc382-38ad-40d4-a51c-350c59903a91\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5ba997ba45d67f36caf2611276438ec7f691f96ca5c0b3f2607b315847813ceb\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2c492f85-691a-4c09-9fea-6f487700d285\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/opt/\", \"title\": \"OPT\", \"tokens\": 2021, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1c03ad42214be3bef44c216a98212b1aa99d710866e7bd9f48360f04a27d01d5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6808, \"end_char_idx\": 7439, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "doc_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "ref_doc_id": "b80bc382-38ad-40d4-a51c-350c59903a91", "chroma:document": "## OPTConfig\n\n[[autodoc]] OPTConfig\n\n<frameworkcontent>\n<pt>\n\n## OPTModel\n\n[[autodoc]] OPTModel\n    - forward\n\n## OPTForCausalLM\n\n[[autodoc]] OPTForCausalLM\n    - forward\n\n## OPTForSequenceClassification\n\n[[autodoc]] OPTForSequenceClassification\n    - forward\n\n## OPTForQuestionAnswering\n\n[[autodoc]] OPTForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFOPTModel\n\n[[autodoc]] TFOPTModel\n    - call\n\n## TFOPTForCausalLM\n\n[[autodoc]] TFOPTForCausalLM\n    - call\n\n</tf>\n<jax>\n\n## FlaxOPTModel\n\n[[autodoc]] FlaxOPTModel\n    - __call__\n\n## FlaxOPTForCausalLM\n\n[[autodoc]] FlaxOPTForCausalLM\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ac43c8e2-4dec-4360-b774-dd6d620a8cf7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5v1.1/", "title": "T5v1.1", "tokens": 789, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ac43c8e2-4dec-4360-b774-dd6d620a8cf7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5v1.1/\", \"title\": \"T5v1.1\", \"tokens\": 789, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b196f1c3-d67d-42de-a3ee-6bd51a9b95df\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5v1.1/\", \"title\": \"T5v1.1\", \"tokens\": 789, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c87fbefb0afa012bb9d37ca43f9bbe4af3073667a24747a55b2da0ac5f067607\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2292, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b196f1c3-d67d-42de-a3ee-6bd51a9b95df", "doc_id": "b196f1c3-d67d-42de-a3ee-6bd51a9b95df", "ref_doc_id": "b196f1c3-d67d-42de-a3ee-6bd51a9b95df", "chroma:document": "# T5v1.1\n\n## Overview\n\nT5v1.1 was released in the [google-research/text-to-text-transfer-transformer](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511)\nrepository by Colin Raffel et al. It's an improved version of the original T5 model.\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be\nfound [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511).\n\n## Usage tips\n\nOne can directly plug in the weights of T5v1.1 into a T5 model, like so:\n\n```python\n>>> from transformers import T5ForConditionalGeneration\n\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google/t5-v1_1-base\")\n```\n\nT5 Version 1.1 includes the following improvements compared to the original T5 model:\n\n- GEGLU activation in the feed-forward hidden layer, rather than ReLU. See [this paper](https://arxiv.org/abs/2002.05202).\n\n- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n\n- Pre-trained on C4 only without mixing in the downstream tasks.\n\n- No parameter sharing between the embedding and classifier layer.\n\n- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller\n  `num_heads` and `d_ff`.\n\nNote: T5 Version 1.1 was only pre-trained on [C4](https://huggingface.co/datasets/c4) excluding any supervised\ntraining. Therefore, this model has to be fine-tuned before it is usable on a downstream task, unlike the original T5\nmodel. Since t5v1.1 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\nGoogle has released the following variants:\n\n- [google/t5-v1_1-small](https://huggingface.co/google/t5-v1_1-small)\n\n- [google/t5-v1_1-base](https://huggingface.co/google/t5-v1_1-base)\n\n- [google/t5-v1_1-large](https://huggingface.co/google/t5-v1_1-large)\n\n- [google/t5-v1_1-xl](https://huggingface.co/google/t5-v1_1-xl)\n\n- [google/t5-v1_1-xxl](https://huggingface.co/google/t5-v1_1-xxl).\n\n\n<Tip>\n\nRefer to [T5's documentation page](t5) for all API reference, tips, code examples and notebooks.\n\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3ef46292-eea2-4a61-9737-c87ad587ab66
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_mae/", "title": "ViTMAE", "tokens": 1730, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3ef46292-eea2-4a61-9737-c87ad587ab66\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"792b57e1-4e5e-4caf-9ca7-c0f226bd0489\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b80ee122d0998a49fbc7a57b86f9e7202ee4eb47f92b0dbf6f0e8ba409fd3418\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f56acf65-7fdb-4944-bada-6290660391e0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fe91e45cfe245fd05511a1ff0059010a9da0ab510af224f47e0654713435778e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3151, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "doc_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "ref_doc_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "chroma:document": "# ViTMAE\n\n## Overview\n\nThe ViTMAE model was proposed in [Masked Autoencoders Are Scalable Vision Learners](https://arxiv.org/abs/2111.06377v2) by Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li,\nPiotr Doll\u00e1r, Ross Girshick. The paper shows that, by pre-training a Vision Transformer (ViT) to reconstruct pixel values for masked patches, one can get results after\nfine-tuning that outperform supervised pre-training.\n\nThe abstract from the paper is the following:\n\n*This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the\ninput image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates\nonly on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask\ntokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs\nenables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity\nmodels that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream\ntasks outperforms supervised pre-training and shows promising scaling behavior.*\n\n<img src=\"https://user-images.githubusercontent.com/11435359/146857310-f258c86c-fde6-48e8-9cee-badd2b21bd2c.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> MAE architecture. Taken from the <a href=\"https://arxiv.org/abs/2111.06377\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [sayakpaul](https://github.com/sayakpaul) and \n[ariG23498](https://github.com/ariG23498) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/mae). \n\n## Usage tips\n\n- MAE (masked auto encoding) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training objective is relatively simple:\nby masking a large portion (75%) of the image patches, the model must reconstruct raw pixel values. One can use [`ViTMAEForPreTraining`] for this purpose.\n- After pre-training, one \"throws away\" the decoder used to reconstruct pixels, and one uses the encoder for fine-tuning/linear probing. This means that after\nfine-tuning, one can directly plug in the weights into a [`ViTForImageClassification`].\n- One can use [`ViTImageProcessor`] to prepare images for the model. See the code examples for more info.\n- Note that the encoder of MAE is only used to encode the visual patches. The encoded patches are then concatenated with mask tokens, which the decoder (which also\nconsists of Transformer blocks) takes as input. Each mask token is a shared, learned vector that indicates the presence of a missing patch to be predicted."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f56acf65-7fdb-4944-bada-6290660391e0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_mae/", "title": "ViTMAE", "tokens": 1730, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f56acf65-7fdb-4944-bada-6290660391e0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"792b57e1-4e5e-4caf-9ca7-c0f226bd0489\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b80ee122d0998a49fbc7a57b86f9e7202ee4eb47f92b0dbf6f0e8ba409fd3418\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3ef46292-eea2-4a61-9737-c87ad587ab66\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"811922d6010a93fb3c06a059ed71ff6809296118153094353a29dd07aecb0c89\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1900277d-c2cb-4e02-996e-ff9f26c386f6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9a142951efa4f276a55ebca0421f76529c6b01de5eb940cae1576029b2ff945d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1752, \"end_char_idx\": 4581, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "doc_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "ref_doc_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2111.06377\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [sayakpaul](https://github.com/sayakpaul) and \n[ariG23498](https://github.com/ariG23498) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/mae). \n\n## Usage tips\n\n- MAE (masked auto encoding) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training objective is relatively simple:\nby masking a large portion (75%) of the image patches, the model must reconstruct raw pixel values. One can use [`ViTMAEForPreTraining`] for this purpose.\n- After pre-training, one \"throws away\" the decoder used to reconstruct pixels, and one uses the encoder for fine-tuning/linear probing. This means that after\nfine-tuning, one can directly plug in the weights into a [`ViTForImageClassification`].\n- One can use [`ViTImageProcessor`] to prepare images for the model. See the code examples for more info.\n- Note that the encoder of MAE is only used to encode the visual patches. The encoded patches are then concatenated with mask tokens, which the decoder (which also\nconsists of Transformer blocks) takes as input. Each mask token is a shared, learned vector that indicates the presence of a missing patch to be predicted. Fixed\nsin/cos position embeddings are added both to the input of the encoder and the decoder.\n- For a visual understanding of how MAEs work you can check out this [post](https://keras.io/examples/vision/masked_image_modeling/).\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTMAEModel\nmodel = ViTMAEModel.from_pretrained(\"facebook/vit-mae-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-mae-base` model, we saw the following speedups during inference."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1900277d-c2cb-4e02-996e-ff9f26c386f6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_mae/", "title": "ViTMAE", "tokens": 1730, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1900277d-c2cb-4e02-996e-ff9f26c386f6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"792b57e1-4e5e-4caf-9ca7-c0f226bd0489\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b80ee122d0998a49fbc7a57b86f9e7202ee4eb47f92b0dbf6f0e8ba409fd3418\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f56acf65-7fdb-4944-bada-6290660391e0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3cbf19312436928473198d2eb0dd9665ef72decd9505effc0de6b008bcde11b5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"37a493f7-89c3-4b54-83a5-8af2bfa4a590\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"29256ad15e3d40cf9dc11e7b8fa1658e32bdfac4227953a4d06fb8d32da1a3db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3044, \"end_char_idx\": 6220, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "doc_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "ref_doc_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "chroma:document": "Each mask token is a shared, learned vector that indicates the presence of a missing patch to be predicted. Fixed\nsin/cos position embeddings are added both to the input of the encoder and the decoder.\n- For a visual understanding of how MAEs work you can check out this [post](https://keras.io/examples/vision/masked_image_modeling/).\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTMAEModel\nmodel = ViTMAEModel.from_pretrained(\"facebook/vit-mae-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-mae-base` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        11 |                                         6 |                      1.83 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViTMAE.\n\n- [`ViTMAEForPreTraining`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining), allowing you to pre-train the model from scratch/further pre-train the model on custom data.\n- A notebook that illustrates how to visualize reconstructed pixel values with [`ViTMAEForPreTraining`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/ViTMAE/ViT_MAE_visualization_demo.ipynb).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 37a493f7-89c3-4b54-83a5-8af2bfa4a590
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_mae/", "title": "ViTMAE", "tokens": 1730, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"37a493f7-89c3-4b54-83a5-8af2bfa4a590\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"792b57e1-4e5e-4caf-9ca7-c0f226bd0489\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b80ee122d0998a49fbc7a57b86f9e7202ee4eb47f92b0dbf6f0e8ba409fd3418\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1900277d-c2cb-4e02-996e-ff9f26c386f6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_mae/\", \"title\": \"ViTMAE\", \"tokens\": 1730, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7c2cfe698df90f88c6d5b79509ce85b8b2f240b090cbc35d0d5142dd7a1b88b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4380, \"end_char_idx\": 6611, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "doc_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "ref_doc_id": "792b57e1-4e5e-4caf-9ca7-c0f226bd0489", "chroma:document": "`torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-mae-base` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        11 |                                         6 |                      1.83 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViTMAE.\n\n- [`ViTMAEForPreTraining`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining), allowing you to pre-train the model from scratch/further pre-train the model on custom data.\n- A notebook that illustrates how to visualize reconstructed pixel values with [`ViTMAEForPreTraining`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/ViTMAE/ViT_MAE_visualization_demo.ipynb).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ViTMAEConfig\n\n[[autodoc]] ViTMAEConfig\n\n<frameworkcontent>\n<pt>\n\n## ViTMAEModel\n\n[[autodoc]] ViTMAEModel\n    - forward\n\n## ViTMAEForPreTraining\n\n[[autodoc]] transformers.ViTMAEForPreTraining\n    - forward\n\n</pt>\n<tf>\n\n## TFViTMAEModel\n\n[[autodoc]] TFViTMAEModel\n    - call\n\n## TFViTMAEForPreTraining\n\n[[autodoc]] transformers.TFViTMAEForPreTraining\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1fa3c11e-4a10-4b10-9e45-72f5fba598ac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ibert/", "title": "I-BERT", "tokens": 822, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1fa3c11e-4a10-4b10-9e45-72f5fba598ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ibert/\", \"title\": \"I-BERT\", \"tokens\": 822, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"727d3d79-a09d-4284-ba39-163f3f5f6851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ibert/\", \"title\": \"I-BERT\", \"tokens\": 822, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8ee176089adea4a60e944013c2386c3032428a16664fcba27919af467acd377e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2741, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "727d3d79-a09d-4284-ba39-163f3f5f6851", "doc_id": "727d3d79-a09d-4284-ba39-163f3f5f6851", "ref_doc_id": "727d3d79-a09d-4284-ba39-163f3f5f6851", "chroma:document": "# I-BERT\n\n## Overview\n\nThe I-BERT model was proposed in [I-BERT: Integer-only BERT Quantization](https://arxiv.org/abs/2101.01321) by\nSehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney and Kurt Keutzer. It's a quantized version of RoBERTa running\ninference up to four times faster.\n\nThe abstract from the paper is the following:\n\n*Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language\nProcessing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive for\nefficient inference at the edge, and even at the data center. While quantization can be a viable solution for this,\nprevious work on quantizing Transformer based models use floating-point arithmetic during inference, which cannot\nefficiently utilize integer-only logical units such as the recent Turing Tensor Cores, or traditional integer-only ARM\nprocessors. In this work, we propose I-BERT, a novel quantization scheme for Transformer based models that quantizes\nthe entire inference with integer-only arithmetic. Based on lightweight integer-only approximation methods for\nnonlinear operations, e.g., GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end integer-only BERT\ninference without any floating point calculation. We evaluate our approach on GLUE downstream tasks using\nRoBERTa-Base/Large. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to\nthe full-precision baseline. Furthermore, our preliminary implementation of I-BERT shows a speedup of 2.4 - 4.0x for\nINT8 inference on a T4 GPU system as compared to FP32 inference. The framework has been developed in PyTorch and has\nbeen open-sourced.*\n\nThis model was contributed by [kssteven](https://huggingface.co/kssteven). The original code can be found [here](https://github.com/kssteven418/I-BERT).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/masked_language_modeling)\n\n## IBertConfig\n\n[[autodoc]] IBertConfig\n\n## IBertModel\n\n[[autodoc]] IBertModel\n    - forward\n\n## IBertForMaskedLM\n\n[[autodoc]] IBertForMaskedLM\n    - forward\n\n## IBertForSequenceClassification\n\n[[autodoc]] IBertForSequenceClassification\n    - forward\n\n## IBertForMultipleChoice\n\n[[autodoc]] IBertForMultipleChoice\n    - forward\n\n## IBertForTokenClassification\n\n[[autodoc]] IBertForTokenClassification\n    - forward\n\n## IBertForQuestionAnswering\n\n[[autodoc]] IBertForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e595c963-7db3-422b-a9b4-ecd454866749
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/decision_transformer/", "title": "Decision Transformer", "tokens": 552, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e595c963-7db3-422b-a9b4-ecd454866749\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/decision_transformer/\", \"title\": \"Decision Transformer\", \"tokens\": 552, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"064f5345-1818-4ca4-afc1-0e5c2b0b0914\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/decision_transformer/\", \"title\": \"Decision Transformer\", \"tokens\": 552, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"472393e44919bd69f1fec34324551236012f36bf128169c3df7709aee9146fff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1799, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "064f5345-1818-4ca4-afc1-0e5c2b0b0914", "doc_id": "064f5345-1818-4ca4-afc1-0e5c2b0b0914", "ref_doc_id": "064f5345-1818-4ca4-afc1-0e5c2b0b0914", "chroma:document": "# Decision Transformer\n\n## Overview\n\nThe Decision Transformer model was proposed in [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://arxiv.org/abs/2106.01345)  \nby Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch.\n\nThe abstract from the paper is the following:\n\n*We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. \nThis allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances\n in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that \n casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or \n compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked \n Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our \n Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, \n Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on \n Atari, OpenAI Gym, and Key-to-Door tasks.*\n\nThis version of the model is for tasks where the state is a vector.\n\nThis model was contributed by [edbeeching](https://huggingface.co/edbeeching). The original code can be found [here](https://github.com/kzl/decision-transformer).\n\n## DecisionTransformerConfig\n\n[[autodoc]] DecisionTransformerConfig\n\n\n## DecisionTransformerGPT2Model\n\n[[autodoc]] DecisionTransformerGPT2Model\n    - forward\n\n## DecisionTransformerModel\n\n[[autodoc]] DecisionTransformerModel\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 212b0f28-c53f-4409-977f-31a52655222f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pegasus/", "title": "Pegasus", "tokens": 1626, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"212b0f28-c53f-4409-977f-31a52655222f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus/\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"45c52fce-ec84-4c0c-9839-d43e787d3414\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus/\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1de092b9f0d72cb8fb9b4e0af00e9e7455a6d4c2604d8e9264d1d81e6395d8f2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bd96b436-3753-4f92-a7f6-a751d5f7f035\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"138e36ec459e0fe7b6f521726551f9dbe65de5ba8be1f6d4b960f855b94e0c12\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3014, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "45c52fce-ec84-4c0c-9839-d43e787d3414", "doc_id": "45c52fce-ec84-4c0c-9839-d43e787d3414", "ref_doc_id": "45c52fce-ec84-4c0c-9839-d43e787d3414", "chroma:document": "# Pegasus\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=pegasus\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-pegasus-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/pegasus_paraphrase\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n\n## Overview\n\nThe Pegasus model was proposed in [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://arxiv.org/pdf/1912.08777.pdf) by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019.\n\nAccording to the abstract,\n\n- Pegasus' pretraining task is intentionally similar to summarization: important sentences are removed/masked from an\n  input document and are generated together as one output sequence from the remaining sentences, similar to an\n  extractive summary.\n- Pegasus achieves SOTA summarization performance on all 12 downstream tasks, as measured by ROUGE and human eval.\n\nThis model was contributed by [sshleifer](https://huggingface.co/sshleifer). The Authors' code can be found [here](https://github.com/google-research/pegasus).\n\n## Usage tips\n\n- Sequence-to-sequence model with the same encoder-decoder model architecture as BART. Pegasus is pre-trained jointly on two self-supervised objective functions: Masked Language Modeling (MLM) and a novel summarization specific pretraining objective, called Gap Sentence Generation (GSG).\n\n  * MLM: encoder input tokens are randomly replaced by a mask tokens and have to be predicted by the encoder (like in BERT)\n  * GSG: whole encoder input sentences are replaced by a second mask token and fed to the decoder, but which has a causal mask to hide the future words like a regular auto-regressive transformer decoder.\n\n- FP16 is not supported (help/ideas on this appreciated!).\n- The adafactor optimizer is recommended for pegasus fine-tuning.\n\n\n## Checkpoints\n\nAll the [checkpoints](https://huggingface.co/models?search=pegasus) are fine-tuned for summarization, besides\n*pegasus-large*, whence the other checkpoints are fine-tuned:\n\n- Each checkpoint is 2.2 GB on disk and 568M parameters.\n- FP16 is not supported (help/ideas on this appreciated!).\n- Summarizing xsum in fp32 takes about 400ms/sample, with default parameters on a v100 GPU.\n- Full replication results and correctly pre-processed data can be found in this [Issue](https://github.com/huggingface/transformers/issues/6844#issue-689259666).\n- [Distilled checkpoints](https://huggingface.co/models?search=distill-pegasus) are described in this [paper](https://arxiv.org/abs/2010.13002).\n\n## Implementation Notes\n\n- All models are transformer encoder-decoders with 16 layers in each component.\n- The implementation is completely inherited from [`BartForConditionalGeneration`]\n- Some key configuration differences:\n  - static, sinusoidal position embeddings\n  - the model starts generating with pad_token_id (which has 0 token_embedding) as the prefix."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: bd96b436-3753-4f92-a7f6-a751d5f7f035
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pegasus/", "title": "Pegasus", "tokens": 1626, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"bd96b436-3753-4f92-a7f6-a751d5f7f035\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus/\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"45c52fce-ec84-4c0c-9839-d43e787d3414\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus/\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1de092b9f0d72cb8fb9b4e0af00e9e7455a6d4c2604d8e9264d1d81e6395d8f2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"212b0f28-c53f-4409-977f-31a52655222f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus/\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9d054603d65ce84a3f92e94345745e92990af00dd4178cc9408d000e5f842952\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fc2fbb28-e009-4b23-bb5e-45e5f6b4f941\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7638b666072aeba6df2afb42d1648974cf72a28cc881b8e864f37a7423966c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1945, \"end_char_idx\": 5136, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "45c52fce-ec84-4c0c-9839-d43e787d3414", "doc_id": "45c52fce-ec84-4c0c-9839-d43e787d3414", "ref_doc_id": "45c52fce-ec84-4c0c-9839-d43e787d3414", "chroma:document": "## Checkpoints\n\nAll the [checkpoints](https://huggingface.co/models?search=pegasus) are fine-tuned for summarization, besides\n*pegasus-large*, whence the other checkpoints are fine-tuned:\n\n- Each checkpoint is 2.2 GB on disk and 568M parameters.\n- FP16 is not supported (help/ideas on this appreciated!).\n- Summarizing xsum in fp32 takes about 400ms/sample, with default parameters on a v100 GPU.\n- Full replication results and correctly pre-processed data can be found in this [Issue](https://github.com/huggingface/transformers/issues/6844#issue-689259666).\n- [Distilled checkpoints](https://huggingface.co/models?search=distill-pegasus) are described in this [paper](https://arxiv.org/abs/2010.13002).\n\n## Implementation Notes\n\n- All models are transformer encoder-decoders with 16 layers in each component.\n- The implementation is completely inherited from [`BartForConditionalGeneration`]\n- Some key configuration differences:\n  - static, sinusoidal position embeddings\n  - the model starts generating with pad_token_id (which has 0 token_embedding) as the prefix.\n  - more beams are used (`num_beams=8`)\n- All pretrained pegasus checkpoints are the same besides three attributes: `tokenizer.model_max_length` (maximum\n  input size), `max_length` (the maximum number of tokens to generate) and `length_penalty`.\n- The code to convert checkpoints trained in the author's [repo](https://github.com/google-research/pegasus) can be\n  found in `convert_pegasus_tf_to_pytorch.py`.\n\n## Usage Example\n\n```python\n>>> from transformers import PegasusForConditionalGeneration, PegasusTokenizer\n>>> import torch\n\n>>> src_text = [\n...     \"\"\" PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.\"\"\"\n... ]\n\n... model_name = \"google/pegasus-xsum\"\n... device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n... tokenizer = PegasusTokenizer.from_pretrained(model_name)\n... model = PegasusForConditionalGeneration.from_pretrained(model_name).to(device)\n... batch = tokenizer(src_text, truncation=True, padding=\"longest\", return_tensors=\"pt\").to(device)\n... translated = model.generate(**batch)\n... tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)\n... assert (\n...     tgt_text[0]\n...     == \"California's largest electricity provider has turned off power to hundreds of thousands of customers.\"\n... )\n```\n\n## Resources\n\n- [Script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/seq2seq-distillation/finetune_pegasus_xsum.sh) to fine-tune pegasus\n  on the XSUM dataset. Data download instructions at [examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md).\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## PegasusConfig\n\n[[autodoc]] PegasusConfig\n\n## PegasusTokenizer\n\nwarning: `add_tokens` does not work at the moment."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fc2fbb28-e009-4b23-bb5e-45e5f6b4f941
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pegasus/", "title": "Pegasus", "tokens": 1626, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fc2fbb28-e009-4b23-bb5e-45e5f6b4f941\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus/\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"45c52fce-ec84-4c0c-9839-d43e787d3414\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus/\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1de092b9f0d72cb8fb9b4e0af00e9e7455a6d4c2604d8e9264d1d81e6395d8f2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bd96b436-3753-4f92-a7f6-a751d5f7f035\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus/\", \"title\": \"Pegasus\", \"tokens\": 1626, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8febfbeaa6cd17e330c7c76a86c790ba4656945856f139c4773446b51f6f401c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3426, \"end_char_idx\": 5900, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "45c52fce-ec84-4c0c-9839-d43e787d3414", "doc_id": "45c52fce-ec84-4c0c-9839-d43e787d3414", "ref_doc_id": "45c52fce-ec84-4c0c-9839-d43e787d3414", "chroma:document": "## Usage Example\n\n```python\n>>> from transformers import PegasusForConditionalGeneration, PegasusTokenizer\n>>> import torch\n\n>>> src_text = [\n...     \"\"\" PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.\"\"\"\n... ]\n\n... model_name = \"google/pegasus-xsum\"\n... device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n... tokenizer = PegasusTokenizer.from_pretrained(model_name)\n... model = PegasusForConditionalGeneration.from_pretrained(model_name).to(device)\n... batch = tokenizer(src_text, truncation=True, padding=\"longest\", return_tensors=\"pt\").to(device)\n... translated = model.generate(**batch)\n... tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)\n... assert (\n...     tgt_text[0]\n...     == \"California's largest electricity provider has turned off power to hundreds of thousands of customers.\"\n... )\n```\n\n## Resources\n\n- [Script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/seq2seq-distillation/finetune_pegasus_xsum.sh) to fine-tune pegasus\n  on the XSUM dataset. Data download instructions at [examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md).\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## PegasusConfig\n\n[[autodoc]] PegasusConfig\n\n## PegasusTokenizer\n\nwarning: `add_tokens` does not work at the moment.\n\n[[autodoc]] PegasusTokenizer\n\n## PegasusTokenizerFast\n\n[[autodoc]] PegasusTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## PegasusModel\n\n[[autodoc]] PegasusModel\n    - forward\n\n## PegasusForConditionalGeneration\n\n[[autodoc]] PegasusForConditionalGeneration\n    - forward\n\n## PegasusForCausalLM\n\n[[autodoc]] PegasusForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFPegasusModel\n\n[[autodoc]] TFPegasusModel\n    - call\n\n## TFPegasusForConditionalGeneration\n\n[[autodoc]] TFPegasusForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxPegasusModel\n\n[[autodoc]] FlaxPegasusModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxPegasusForConditionalGeneration\n\n[[autodoc]] FlaxPegasusForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3fdca2d6-d698-48a9-8246-3950be3fac97
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/poolformer/", "title": "PoolFormer", "tokens": 1298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3fdca2d6-d698-48a9-8246-3950be3fac97\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer/\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b956c414-e981-4ecf-a0cc-23b50a9ffac3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer/\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"cdf3fcda830d5b19e00c22d639ae2d07b9820e63b5019f718e995510f6cbc5f4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7856fe02-1f56-482c-81eb-a0af9a8bb682\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b5c9d53c9b53be72d77863736dc2b0db5705437b1f7c26c17f3ff9718e0c046\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2901, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b956c414-e981-4ecf-a0cc-23b50a9ffac3", "doc_id": "b956c414-e981-4ecf-a0cc-23b50a9ffac3", "ref_doc_id": "b956c414-e981-4ecf-a0cc-23b50a9ffac3", "chroma:document": "# PoolFormer\n\n## Overview\n\nThe PoolFormer model was proposed in [MetaFormer is Actually What You Need for Vision](https://arxiv.org/abs/2111.11418)  by Sea AI Labs. Instead of designing complicated token mixer to achieve SOTA performance, the target of this work is to demonstrate the competence of transformer models largely stem from the general architecture MetaFormer.\n\nThe abstract from the paper is the following:\n\n*Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in transformers with an embarrassingly simple spatial pooling operator to conduct only the most basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned vision transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 48%/60% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of \"MetaFormer\", a general architecture abstracted from transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design.*\n\nThe figure below illustrates the architecture of PoolFormer. Taken from the [original paper](https://arxiv.org/abs/2111.11418).\n\n<img width=\"600\" src=\"https://user-images.githubusercontent.com/15921929/142746124-1ab7635d-2536-4a0e-ad43-b4fe2c5a525d.png\"/>\n\nThis model was contributed by [heytanay](https://huggingface.co/heytanay). The original code can be found [here](https://github.com/sail-sg/poolformer).\n\n## Usage tips\n\n- PoolFormer has a hierarchical architecture, where instead of Attention, a simple Average Pooling layer is present. All checkpoints of the model can be found on the [hub](https://huggingface.co/models?other=poolformer).\n- One can use [`PoolFormerImageProcessor`] to prepare images for the model.\n- As most models, PoolFormer comes in different sizes, the details of which can be found in the table below."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7856fe02-1f56-482c-81eb-a0af9a8bb682
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/poolformer/", "title": "PoolFormer", "tokens": 1298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7856fe02-1f56-482c-81eb-a0af9a8bb682\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer/\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b956c414-e981-4ecf-a0cc-23b50a9ffac3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer/\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"cdf3fcda830d5b19e00c22d639ae2d07b9820e63b5019f718e995510f6cbc5f4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3fdca2d6-d698-48a9-8246-3950be3fac97\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer/\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b21c953276c4cfec4badaead3796b8d93621864a1bb501f4528dbad4d7a7a184\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9630aa56-b275-40a6-b932-712fa6268370\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a2ddb5b7b3030213d9b235a50b519c68ffc8d595070ea28c851d660df2096a43\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1237, \"end_char_idx\": 4083, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b956c414-e981-4ecf-a0cc-23b50a9ffac3", "doc_id": "b956c414-e981-4ecf-a0cc-23b50a9ffac3", "ref_doc_id": "b956c414-e981-4ecf-a0cc-23b50a9ffac3", "chroma:document": "For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned vision transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 48%/60% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of \"MetaFormer\", a general architecture abstracted from transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design.*\n\nThe figure below illustrates the architecture of PoolFormer. Taken from the [original paper](https://arxiv.org/abs/2111.11418).\n\n<img width=\"600\" src=\"https://user-images.githubusercontent.com/15921929/142746124-1ab7635d-2536-4a0e-ad43-b4fe2c5a525d.png\"/>\n\nThis model was contributed by [heytanay](https://huggingface.co/heytanay). The original code can be found [here](https://github.com/sail-sg/poolformer).\n\n## Usage tips\n\n- PoolFormer has a hierarchical architecture, where instead of Attention, a simple Average Pooling layer is present. All checkpoints of the model can be found on the [hub](https://huggingface.co/models?other=poolformer).\n- One can use [`PoolFormerImageProcessor`] to prepare images for the model.\n- As most models, PoolFormer comes in different sizes, the details of which can be found in the table below.\n\n| **Model variant** | **Depths**    | **Hidden sizes**    | **Params (M)** | **ImageNet-1k Top 1** |\n| :---------------: | ------------- | ------------------- | :------------: | :-------------------: |\n| s12               | [2, 2, 6, 2]  | [64, 128, 320, 512] | 12             | 77.2                  |\n| s24               | [4, 4, 12, 4] | [64, 128, 320, 512] | 21             | 80.3                  |\n| s36               | [6, 6, 18, 6] | [64, 128, 320, 512] | 31             | 81.4                  |\n| m36               | [6, 6, 18, 6] | [96, 192, 384, 768] | 56             | 82.1                  |\n| m48               | [8, 8, 24, 8] | [96, 192, 384, 768] | 73             | 82.5                  |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with PoolFormer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`PoolFormerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9630aa56-b275-40a6-b932-712fa6268370
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/poolformer/", "title": "PoolFormer", "tokens": 1298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9630aa56-b275-40a6-b932-712fa6268370\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer/\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b956c414-e981-4ecf-a0cc-23b50a9ffac3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer/\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"cdf3fcda830d5b19e00c22d639ae2d07b9820e63b5019f718e995510f6cbc5f4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7856fe02-1f56-482c-81eb-a0af9a8bb682\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/poolformer/\", \"title\": \"PoolFormer\", \"tokens\": 1298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b5143144b0d6537459405d038ade39d0ef451c527ea99ca20d76344001ed9de1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2793, \"end_char_idx\": 4769, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b956c414-e981-4ecf-a0cc-23b50a9ffac3", "doc_id": "b956c414-e981-4ecf-a0cc-23b50a9ffac3", "ref_doc_id": "b956c414-e981-4ecf-a0cc-23b50a9ffac3", "chroma:document": "- As most models, PoolFormer comes in different sizes, the details of which can be found in the table below.\n\n| **Model variant** | **Depths**    | **Hidden sizes**    | **Params (M)** | **ImageNet-1k Top 1** |\n| :---------------: | ------------- | ------------------- | :------------: | :-------------------: |\n| s12               | [2, 2, 6, 2]  | [64, 128, 320, 512] | 12             | 77.2                  |\n| s24               | [4, 4, 12, 4] | [64, 128, 320, 512] | 21             | 80.3                  |\n| s36               | [6, 6, 18, 6] | [64, 128, 320, 512] | 31             | 81.4                  |\n| m36               | [6, 6, 18, 6] | [96, 192, 384, 768] | 56             | 82.1                  |\n| m48               | [8, 8, 24, 8] | [96, 192, 384, 768] | 73             | 82.5                  |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with PoolFormer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`PoolFormerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## PoolFormerConfig\n\n[[autodoc]] PoolFormerConfig\n\n## PoolFormerFeatureExtractor\n\n[[autodoc]] PoolFormerFeatureExtractor\n    - __call__\n\n## PoolFormerImageProcessor\n\n[[autodoc]] PoolFormerImageProcessor\n    - preprocess\n\n## PoolFormerModel\n\n[[autodoc]] PoolFormerModel\n    - forward\n\n## PoolFormerForImageClassification\n\n[[autodoc]] PoolFormerForImageClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a43ebaa3-a6a1-41c5-88d8-19ecd7dea726
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon/", "title": "Chameleon", "tokens": 2026, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a43ebaa3-a6a1-41c5-88d8-19ecd7dea726\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"da74db71-97c2-4eba-b464-78e342a1f94c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"635341d00cd77982f08d889f0a4e68775c41a70ac7d798a8c1d304f07637700d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"610b168e-770d-4d60-a444-98c67bf8eef5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59230c6179a8780bd62a1ba21ec0d14d2352aef58d3bd044d41cf5eddebc1b6a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3466, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "ref_doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "chroma:document": "# Chameleon\n\n## Overview\n\nThe Chameleon model was proposed in [Chameleon: Mixed-Modal Early-Fusion Foundation Models\n](https://arxiv.org/abs/2405.09818v1) by META AI Chameleon Team. Chameleon is a Vision-Language Model that use vector quantization to tokenize images which enables the model to generate multimodal output. The model takes images and texts as input, including an interleaved format, and generates textual response. Image generation module is not released yet. \n\n\nThe abstract from the paper is the following:\n\n*We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training\napproach from inception, an alignment recipe, and an architectural parameterization tailored for the\nearly-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range\nof tasks, including visual question answering, image captioning, text generation, image generation, and\nlong-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including\nstate-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while\nbeing competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image\ngeneration, all in a single model. It also matches or exceeds the performance of much larger models,\nincluding Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal\ngeneration evaluation, where either the prompt or outputs contain mixed sequences of both images and\ntext. Chameleon marks a significant step forward in unified modeling of full multimodal documents*\n\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/chameleon_arch.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Chameleon incorporates a vector quantizer module to transform images into discrete tokens. That also enables image generation using an auto-regressive transformer. Taken from the <a href=\"https://arxiv.org/abs/2405.09818v1\">original paper.</a> </small>\n\nThis model was contributed by [joaogante](https://huggingface.co/joaogante) and [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).\nThe original code can be found [here](https://github.com/facebookresearch/chameleon).\n\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to set `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note that Chameleon was tuned for safety alignment. If the model is refusing to answer, consider asking a more concrete question, instead of an open question.\n\n- Chameleon generates in chat format which means that the generated text will always be the \"assistant's turn\". You can enable a text completion generation by passing `return_for_text_completion=True` when calling the processor.\n\n> [!NOTE]\n> Chameleon implementation in Transformers uses a special image token to indicate where to merge image embeddings. For special image token we didn't add a new one but used one of the reserved tokens: `<reserved08707>`. You have to add `<image>` to your prompt in the place where the image should be embedded for correct generation.\n\n## Usage example\n\n### Single image inference\n\nChameleon is a gated model so make sure to have access and login to Hugging Face Hub using a token."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 610b168e-770d-4d60-a444-98c67bf8eef5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon/", "title": "Chameleon", "tokens": 2026, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"610b168e-770d-4d60-a444-98c67bf8eef5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"da74db71-97c2-4eba-b464-78e342a1f94c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"635341d00cd77982f08d889f0a4e68775c41a70ac7d798a8c1d304f07637700d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a43ebaa3-a6a1-41c5-88d8-19ecd7dea726\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b20dcbcf34179e6df617daab26dd74505af050bf08b49f6457d87ca1b85a47b5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"602e7570-ecac-495e-aa3b-860ae56f8792\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5fbf2ffc5190d38f41ae62d879e739a118ad5f5e171dfe991c771353321b7129\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1713, \"end_char_idx\": 4509, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "ref_doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "chroma:document": "<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/chameleon_arch.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Chameleon incorporates a vector quantizer module to transform images into discrete tokens. That also enables image generation using an auto-regressive transformer. Taken from the <a href=\"https://arxiv.org/abs/2405.09818v1\">original paper.</a> </small>\n\nThis model was contributed by [joaogante](https://huggingface.co/joaogante) and [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).\nThe original code can be found [here](https://github.com/facebookresearch/chameleon).\n\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to set `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note that Chameleon was tuned for safety alignment. If the model is refusing to answer, consider asking a more concrete question, instead of an open question.\n\n- Chameleon generates in chat format which means that the generated text will always be the \"assistant's turn\". You can enable a text completion generation by passing `return_for_text_completion=True` when calling the processor.\n\n> [!NOTE]\n> Chameleon implementation in Transformers uses a special image token to indicate where to merge image embeddings. For special image token we didn't add a new one but used one of the reserved tokens: `<reserved08707>`. You have to add `<image>` to your prompt in the place where the image should be embedded for correct generation.\n\n## Usage example\n\n### Single image inference\n\nChameleon is a gated model so make sure to have access and login to Hugging Face Hub using a token. \nHere's how to load the model and perform inference in half-precision (`torch.bfloat16`):\n\n```python\nfrom transformers import ChameleonProcessor, ChameleonForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = ChameleonProcessor.from_pretrained(\"facebook/chameleon-7b\")\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", torch_dtype=torch.bfloat16, device_map=\"cuda\")\n\n# prepare image and text prompt\nurl = 'http://images.cocodataset.org/val2017/000000039769.jpg'\nimage = Image.open(requests.get(url, stream=True).raw)\nprompt = \"What do you see in this image?<image>\"\n\ninputs = processor(prompt, image, return_tensors=\"pt\").to(model.device)\n\n# autoregressively complete prompt\noutput = model.generate(**inputs, max_new_tokens=50)\nprint(processor.decode(output[0], skip_special_tokens=True))\n```\n\n### Multi image inference\n\nChameleon can perform inference with multiple images as input, where images either belong to the same prompt or different prompts (in batched inference)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 602e7570-ecac-495e-aa3b-860ae56f8792
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon/", "title": "Chameleon", "tokens": 2026, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"602e7570-ecac-495e-aa3b-860ae56f8792\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"da74db71-97c2-4eba-b464-78e342a1f94c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"635341d00cd77982f08d889f0a4e68775c41a70ac7d798a8c1d304f07637700d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"610b168e-770d-4d60-a444-98c67bf8eef5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f1a5c63fb42263743c9e838ee20c0714d5cd54cacb9f09c835b8a4b89dd1180e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0b454efc-5d37-426b-8394-b44c45445241\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4c4ba360a4c560972e9515e51a109ca5d6c16d8304c39386d36bde209c8b4051\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2748, \"end_char_idx\": 5512, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "ref_doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "chroma:document": "- Chameleon generates in chat format which means that the generated text will always be the \"assistant's turn\". You can enable a text completion generation by passing `return_for_text_completion=True` when calling the processor.\n\n> [!NOTE]\n> Chameleon implementation in Transformers uses a special image token to indicate where to merge image embeddings. For special image token we didn't add a new one but used one of the reserved tokens: `<reserved08707>`. You have to add `<image>` to your prompt in the place where the image should be embedded for correct generation.\n\n## Usage example\n\n### Single image inference\n\nChameleon is a gated model so make sure to have access and login to Hugging Face Hub using a token. \nHere's how to load the model and perform inference in half-precision (`torch.bfloat16`):\n\n```python\nfrom transformers import ChameleonProcessor, ChameleonForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = ChameleonProcessor.from_pretrained(\"facebook/chameleon-7b\")\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", torch_dtype=torch.bfloat16, device_map=\"cuda\")\n\n# prepare image and text prompt\nurl = 'http://images.cocodataset.org/val2017/000000039769.jpg'\nimage = Image.open(requests.get(url, stream=True).raw)\nprompt = \"What do you see in this image?<image>\"\n\ninputs = processor(prompt, image, return_tensors=\"pt\").to(model.device)\n\n# autoregressively complete prompt\noutput = model.generate(**inputs, max_new_tokens=50)\nprint(processor.decode(output[0], skip_special_tokens=True))\n```\n\n### Multi image inference\n\nChameleon can perform inference with multiple images as input, where images either belong to the same prompt or different prompts (in batched inference). Here is how you can do it:\n\n```python\nfrom transformers import ChameleonProcessor, ChameleonForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = ChameleonProcessor.from_pretrained(\"facebook/chameleon-7b\")\n\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", torch_dtype=torch.bfloat16, device_map=\"cuda\")\n\n# Get three different images\nurl = \"https://www.ilankelman.org/stopsigns/australia.jpg\"\nimage_stop = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage_cats = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\nimage_snowman = Image.open(requests.get(url, stream=True).raw)\n\n# Prepare a batched prompt, where the first one is a multi-image prompt and the second is not\nprompts = [\n    \"What do these images have in common?<image><image>\",\n    \"<image>What is shown in this image?\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0b454efc-5d37-426b-8394-b44c45445241
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon/", "title": "Chameleon", "tokens": 2026, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0b454efc-5d37-426b-8394-b44c45445241\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"da74db71-97c2-4eba-b464-78e342a1f94c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"635341d00cd77982f08d889f0a4e68775c41a70ac7d798a8c1d304f07637700d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"602e7570-ecac-495e-aa3b-860ae56f8792\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"546e18016bc097fbd19ea0d7c4bfd0eb7cb9dd440029a40ccdce7365d46b84c7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e74a521d-0656-43c5-88f8-c722d0340d03\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"03e2f7ab4ad2d31b80953193056a7b8870070c58340dffcc5e301915e384fd31\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4510, \"end_char_idx\": 7369, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "ref_doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "chroma:document": "Here is how you can do it:\n\n```python\nfrom transformers import ChameleonProcessor, ChameleonForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = ChameleonProcessor.from_pretrained(\"facebook/chameleon-7b\")\n\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", torch_dtype=torch.bfloat16, device_map=\"cuda\")\n\n# Get three different images\nurl = \"https://www.ilankelman.org/stopsigns/australia.jpg\"\nimage_stop = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage_cats = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\nimage_snowman = Image.open(requests.get(url, stream=True).raw)\n\n# Prepare a batched prompt, where the first one is a multi-image prompt and the second is not\nprompts = [\n    \"What do these images have in common?<image><image>\",\n    \"<image>What is shown in this image?\"\n]\n\n# We can simply feed images in the order they have to be used in the text prompt\n# Each \"<image>\" token uses one image leaving the next for the subsequent \"<image>\" tokens\ninputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors=\"pt\").to(device=\"cuda\", dtype=torch.bfloat16)\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_new_tokens=50)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes\n\nThe model can be loaded in 8 or 4 bits, greatly reducing the memory requirements while maintaining the performance of the original model. First make sure to install bitsandbytes, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:\n\n```python\nfrom transformers import ChameleonForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", quantization_config=quantization_config, device_map=\"cuda\")\n```\n\n### Use Flash-Attention 2 and SDPA to further speed-up generation\n\nThe models supports both, Flash-Attention 2 and PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) which can be enables for optimization. SDPA is the default options when you load the model, If you want to switch for Flash Attention 2, first make sure to install flash-attn. Refer to the [original repository](https://github.com/Dao-AILab/flash-attention) regarding that package installation."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e74a521d-0656-43c5-88f8-c722d0340d03
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chameleon/", "title": "Chameleon", "tokens": 2026, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e74a521d-0656-43c5-88f8-c722d0340d03\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"da74db71-97c2-4eba-b464-78e342a1f94c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"635341d00cd77982f08d889f0a4e68775c41a70ac7d798a8c1d304f07637700d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0b454efc-5d37-426b-8394-b44c45445241\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chameleon/\", \"title\": \"Chameleon\", \"tokens\": 2026, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f86301a8362b2f875dcc95f5e32707ab66b6d7d3c073f2e4e32f4d0393fd00ce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6217, \"end_char_idx\": 8170, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "ref_doc_id": "da74db71-97c2-4eba-b464-78e342a1f94c", "chroma:document": "First make sure to install bitsandbytes, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:\n\n```python\nfrom transformers import ChameleonForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = ChameleonForConditionalGeneration.from_pretrained(\"facebook/chameleon-7b\", quantization_config=quantization_config, device_map=\"cuda\")\n```\n\n### Use Flash-Attention 2 and SDPA to further speed-up generation\n\nThe models supports both, Flash-Attention 2 and PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) which can be enables for optimization. SDPA is the default options when you load the model, If you want to switch for Flash Attention 2, first make sure to install flash-attn. Refer to the [original repository](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with:\n\n```python\nfrom transformers import ChameleonForConditionalGeneration\n\nmodel_id = \"facebook/chameleon-7b\"\nmodel = ChameleonForConditionalGeneration.from_pretrained(\n    model_id, \n    torch_dtype=torch.bfloat16, \n    low_cpu_mem_usage=True,\n    attn_implementation=\"flash_attention_2\"\n).to(0)\n```\n\n## ChameleonConfig\n\n[[autodoc]] ChameleonConfig\n\n## ChameleonVQVAEConfig\n\n[[autodoc]] ChameleonVQVAEConfig\n\n## ChameleonProcessor\n\n[[autodoc]] ChameleonProcessor\n\n## ChameleonImageProcessor\n\n[[autodoc]] ChameleonImageProcessor\n    - preprocess\n\n## ChameleonVQVAE\n\n[[autodoc]] ChameleonVQVAE\n    - forward\n\n## ChameleonModel\n\n[[autodoc]] ChameleonModel\n    - forward\n\n## ChameleonForConditionalGeneration\n\n[[autodoc]] ChameleonForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 407554b8-60da-4101-90dc-b5c7b9d03817
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yoso/", "title": "YOSO", "tokens": 1066, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"407554b8-60da-4101-90dc-b5c7b9d03817\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso/\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"74c0495e-f99c-4e35-a8ad-33fefb20c004\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso/\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0e08fb43c229db3b2c3cf40b22de741fc24e58e6b3039696737650f2297995a9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2ca3730e-713c-455e-9750-557dc442a00e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4cdce73741b35c392b7e3d6fbab67bfbb79d635c475503899c19d9ab5106faf0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2986, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "74c0495e-f99c-4e35-a8ad-33fefb20c004", "doc_id": "74c0495e-f99c-4e35-a8ad-33fefb20c004", "ref_doc_id": "74c0495e-f99c-4e35-a8ad-33fefb20c004", "chroma:document": "# YOSO\n\n## Overview\n\nThe YOSO model was proposed in [You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling](https://arxiv.org/abs/2111.09714)  \nby Zhanpeng Zeng, Yunyang Xiong, Sathya N. Ravi, Shailesh Acharya, Glenn Fung, Vikas Singh. YOSO approximates standard softmax self-attention\nvia a Bernoulli sampling scheme based on Locality Sensitive Hashing (LSH). In principle, all the Bernoulli random variables can be sampled with\na single hash. \n\nThe abstract from the paper is the following:\n\n*Transformer-based models are widely used in natural language processing (NLP). Central to the transformer model is \nthe self-attention mechanism, which captures the interactions of token pairs in the input sequences and depends quadratically \non the sequence length. Training such models on longer sequences is expensive. In this paper, we show that a Bernoulli sampling \nattention mechanism based on Locality Sensitive Hashing (LSH), decreases the quadratic complexity of such models to linear. \nWe bypass the quadratic cost by considering self-attention as a sum of individual tokens associated with Bernoulli random \nvariables that can, in principle, be sampled at once by a single hash (although in practice, this number may be a small constant). \nThis leads to an efficient sampling scheme to estimate self-attention which relies on specific modifications of \nLSH (to enable deployment on GPU architectures). We evaluate our algorithm on the GLUE benchmark with standard 512 sequence \nlength where we see favorable performance relative to a standard pretrained Transformer. On the Long Range Arena (LRA) benchmark, \nfor evaluating performance on long sequences, our method achieves results consistent with softmax self-attention but with sizable \nspeed-ups and memory savings and often outperforms other efficient self-attention methods. Our code is available at this https URL*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The original code can be found [here](https://github.com/mlpen/YOSO).\n\n## Usage tips\n\n- The YOSO attention algorithm is implemented through custom CUDA kernels, functions written in CUDA C++ that can be executed multiple times\nin parallel on a GPU.\n- The kernels provide a `fast_hash` function, which approximates the random projections of the queries and keys using the Fast Hadamard Transform. Using these\nhash codes, the `lsh_cumulation` function approximates self-attention via LSH-based Bernoulli sampling.\n- To use the custom kernels, the user should set `config.use_expectation = False`. To ensure that the kernels are compiled successfully, \nthe user must install the correct version of PyTorch and cudatoolkit. By default, `config.use_expectation = True`, which uses YOSO-E and \ndoes not require compiling CUDA kernels.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/yoso_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> YOSO Attention Algorithm."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2ca3730e-713c-455e-9750-557dc442a00e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yoso/", "title": "YOSO", "tokens": 1066, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2ca3730e-713c-455e-9750-557dc442a00e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso/\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"74c0495e-f99c-4e35-a8ad-33fefb20c004\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso/\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0e08fb43c229db3b2c3cf40b22de741fc24e58e6b3039696737650f2297995a9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"407554b8-60da-4101-90dc-b5c7b9d03817\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yoso/\", \"title\": \"YOSO\", \"tokens\": 1066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"59961570321b788cd24ad1b7627c4457505d3a6c1fc8b0b91557638ff1820311\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1274, \"end_char_idx\": 3903, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "74c0495e-f99c-4e35-a8ad-33fefb20c004", "doc_id": "74c0495e-f99c-4e35-a8ad-33fefb20c004", "ref_doc_id": "74c0495e-f99c-4e35-a8ad-33fefb20c004", "chroma:document": "This leads to an efficient sampling scheme to estimate self-attention which relies on specific modifications of \nLSH (to enable deployment on GPU architectures). We evaluate our algorithm on the GLUE benchmark with standard 512 sequence \nlength where we see favorable performance relative to a standard pretrained Transformer. On the Long Range Arena (LRA) benchmark, \nfor evaluating performance on long sequences, our method achieves results consistent with softmax self-attention but with sizable \nspeed-ups and memory savings and often outperforms other efficient self-attention methods. Our code is available at this https URL*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The original code can be found [here](https://github.com/mlpen/YOSO).\n\n## Usage tips\n\n- The YOSO attention algorithm is implemented through custom CUDA kernels, functions written in CUDA C++ that can be executed multiple times\nin parallel on a GPU.\n- The kernels provide a `fast_hash` function, which approximates the random projections of the queries and keys using the Fast Hadamard Transform. Using these\nhash codes, the `lsh_cumulation` function approximates self-attention via LSH-based Bernoulli sampling.\n- To use the custom kernels, the user should set `config.use_expectation = False`. To ensure that the kernels are compiled successfully, \nthe user must install the correct version of PyTorch and cudatoolkit. By default, `config.use_expectation = True`, which uses YOSO-E and \ndoes not require compiling CUDA kernels.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/yoso_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> YOSO Attention Algorithm. Taken from the <a href=\"https://arxiv.org/abs/2111.09714\">original paper</a>.</small>\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## YosoConfig\n\n[[autodoc]] YosoConfig\n\n## YosoModel\n\n[[autodoc]] YosoModel\n    - forward\n\n## YosoForMaskedLM\n\n[[autodoc]] YosoForMaskedLM\n    - forward\n\n## YosoForSequenceClassification\n\n[[autodoc]] YosoForSequenceClassification\n    - forward\n\n## YosoForMultipleChoice\n\n[[autodoc]] YosoForMultipleChoice\n    - forward\n\n## YosoForTokenClassification\n\n[[autodoc]] YosoForTokenClassification\n    - forward\n\n## YosoForQuestionAnswering\n\n[[autodoc]] YosoForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 39f764d8-8f93-44c3-b2a7-0d99546e1365
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trajectory_transformer/", "title": "Trajectory Transformer", "tokens": 686, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"39f764d8-8f93-44c3-b2a7-0d99546e1365\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trajectory_transformer/\", \"title\": \"Trajectory Transformer\", \"tokens\": 686, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e32eedd-13b9-4c88-9517-561ce4adae6b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trajectory_transformer/\", \"title\": \"Trajectory Transformer\", \"tokens\": 686, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5554c2b71b2a0c947ba9158b2f1156d4d0a057f24a2481c49aeeaa8435fc949d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2444, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e32eedd-13b9-4c88-9517-561ce4adae6b", "doc_id": "4e32eedd-13b9-4c88-9517-561ce4adae6b", "ref_doc_id": "4e32eedd-13b9-4c88-9517-561ce4adae6b", "chroma:document": "# Trajectory Transformer\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, so we won't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe Trajectory Transformer model was proposed in [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://arxiv.org/abs/2106.02039)  by Michael Janner, Qiyang Li, Sergey Levine.\n\nThe abstract from the paper is the following:\n\n*Reinforcement learning (RL) is typically concerned with estimating stationary policies or single-step models,\nleveraging the Markov property to factorize problems in time. However, we can also view RL as a generic sequence\nmodeling problem, with the goal being to produce a sequence of actions that leads to a sequence of high rewards.\nViewed in this way, it is tempting to consider whether high-capacity sequence prediction models that work well\nin other domains, such as natural-language processing, can also provide effective solutions to the RL problem.\nTo this end, we explore how RL can be tackled with the tools of sequence modeling, using a Transformer architecture\nto model distributions over trajectories and repurposing beam search as a planning algorithm. Framing RL as sequence\nmodeling problem simplifies a range of design decisions, allowing us to dispense with many of the components common\nin offline RL algorithms. We demonstrate the flexibility of this approach across long-horizon dynamics prediction,\nimitation learning, goal-conditioned RL, and offline RL. Further, we show that this approach can be combined with\nexisting model-free algorithms to yield a state-of-the-art planner in sparse-reward, long-horizon tasks.*\n\nThis model was contributed by [CarlCochet](https://huggingface.co/CarlCochet). The original code can be found [here](https://github.com/jannerm/trajectory-transformer).\n\n## Usage tips\n\nThis Transformer is used for deep reinforcement learning. To use it, you need to create sequences from\nactions, states and rewards from all previous timesteps. This model will treat all these elements together\nas one big sequence (a trajectory).\n\n## TrajectoryTransformerConfig\n\n[[autodoc]] TrajectoryTransformerConfig\n\n## TrajectoryTransformerModel\n\n[[autodoc]] TrajectoryTransformerModel\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b7dd30b3-e114-4010-86bf-f9f9e950528f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/stablelm/", "title": "StableLM", "tokens": 1230, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b7dd30b3-e114-4010-86bf-f9f9e950528f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm/\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c47450d3-61d0-4fa6-86c2-908c85275ed1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm/\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"af485d20e10666f3d618a8022b0a9ac0391b29976f14218ee1e5ddbe3cd506c8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"42703c18-0eb2-4f93-a20f-ce9694162ad7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9bf9af452191f5c956f24d5a6f0a7709ac153d838c96d677c68642a3f30968e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2504, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c47450d3-61d0-4fa6-86c2-908c85275ed1", "doc_id": "c47450d3-61d0-4fa6-86c2-908c85275ed1", "ref_doc_id": "c47450d3-61d0-4fa6-86c2-908c85275ed1", "chroma:document": "# StableLM\n\n## Overview\n\n`StableLM 3B 4E1T` was proposed in [`StableLM 3B 4E1T`: Technical Report](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo) by Stability AI and is the first model in a series of multi-epoch pre-trained language models.\n\n### Model Details\n\n`StableLM 3B 4E1T` is a decoder-only base language model pre-trained on 1 trillion tokens of diverse English and code datasets for four epochs.\nThe model architecture is transformer-based with partial Rotary Position Embeddings, SwiGLU activation, LayerNorm, etc.\n\nWe also provide `StableLM Zephyr 3B`, an instruction fine-tuned version of the model that can be used for chat-based applications.\n\n### Usage Tips\n\n- The architecture is similar to LLaMA but with RoPE applied to 25% of head embedding dimensions, LayerNorm instead of RMSNorm, and optional QKV bias terms.\n- `StableLM 3B 4E1T`-based models uses the same tokenizer as [`GPTNeoXTokenizerFast`].\n\n`StableLM 3B 4E1T` and `StableLM Zephyr 3B` can be found on the [Huggingface Hub](https://huggingface.co/stabilityai)\n\nThe following code snippet demonstrates how to use `StableLM 3B 4E1T` for inference:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> set_seed(0)\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> model_inputs = tokenizer(\"The weather is always wonderful in\", return_tensors=\"pt\").to(model.device)\n\n>>> generated_ids = model.generate(**model_inputs, max_length=32, do_sample=True)\n>>> responses = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n>>> responses\n['The weather is always wonderful in Costa Rica, which makes it a prime destination for retirees. That\u2019s where the Pensionado program comes in, offering']\n```\n\n## Combining StableLM and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention v2.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nAlso make sure that your hardware is compatible with Flash-Attention 2. Read more about it in the official documentation of the [`flash-attn`](https://github.com/Dao-AILab/flash-attention) repository. Note: you must load your model in half-precision (e.g. `torch.bfloat16`)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 42703c18-0eb2-4f93-a20f-ce9694162ad7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/stablelm/", "title": "StableLM", "tokens": 1230, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"42703c18-0eb2-4f93-a20f-ce9694162ad7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm/\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c47450d3-61d0-4fa6-86c2-908c85275ed1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm/\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"af485d20e10666f3d618a8022b0a9ac0391b29976f14218ee1e5ddbe3cd506c8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7dd30b3-e114-4010-86bf-f9f9e950528f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/stablelm/\", \"title\": \"StableLM\", \"tokens\": 1230, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b63607ae81b9fe954777d43b494a629db88d93760687a73dfffb4044d7843d7e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1040, \"end_char_idx\": 3907, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c47450d3-61d0-4fa6-86c2-908c85275ed1", "doc_id": "c47450d3-61d0-4fa6-86c2-908c85275ed1", "ref_doc_id": "c47450d3-61d0-4fa6-86c2-908c85275ed1", "chroma:document": "`StableLM 3B 4E1T` and `StableLM Zephyr 3B` can be found on the [Huggingface Hub](https://huggingface.co/stabilityai)\n\nThe following code snippet demonstrates how to use `StableLM 3B 4E1T` for inference:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> set_seed(0)\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> model_inputs = tokenizer(\"The weather is always wonderful in\", return_tensors=\"pt\").to(model.device)\n\n>>> generated_ids = model.generate(**model_inputs, max_length=32, do_sample=True)\n>>> responses = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n>>> responses\n['The weather is always wonderful in Costa Rica, which makes it a prime destination for retirees. That\u2019s where the Pensionado program comes in, offering']\n```\n\n## Combining StableLM and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention v2.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nAlso make sure that your hardware is compatible with Flash-Attention 2. Read more about it in the official documentation of the [`flash-attn`](https://github.com/Dao-AILab/flash-attention) repository. Note: you must load your model in half-precision (e.g. `torch.bfloat16`).\n\nNow, to run the model with Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> set_seed(0)\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"stabilityai/stablelm-3b-4e1t\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"stabilityai/stablelm-3b-4e1t\", torch_dtype=torch.bfloat16, attn_implementation=\"flash_attention_2\")  # doctest: +SKIP\n>>> model.to(device)  # doctest: +SKIP\n\n>>> model_inputs = tokenizer(\"The weather is always wonderful in\", return_tensors=\"pt\").to(model.device)\n\n>>> generated_ids = model.generate(**model_inputs, max_length=32, do_sample=True)  # doctest: +SKIP\n>>> responses = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)  # doctest: +SKIP\n>>> responses  # doctest: +SKIP\n['The weather is always wonderful in Costa Rica, which makes it a prime destination for retirees. That\u2019s where the Pensionado program comes in, offering']\n```\n\n\n## StableLmConfig\n\n[[autodoc]] StableLmConfig\n\n## StableLmModel\n\n[[autodoc]] StableLmModel\n    - forward\n\n## StableLmForCausalLM\n\n[[autodoc]] StableLmForCausalLM\n    - forward\n\n## StableLmForSequenceClassification\n\n[[autodoc]] StableLmForSequenceClassification\n    - forward\n\n## StableLmForTokenClassification\n\n[[autodoc]] StableLmForTokenClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 959b308d-26c6-41a2-bc51-787db870aad4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bertweet/", "title": "BERTweet", "tokens": 666, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"959b308d-26c6-41a2-bc51-787db870aad4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bertweet/\", \"title\": \"BERTweet\", \"tokens\": 666, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a17a6eff-44aa-4573-85c5-45d52c18f161\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bertweet/\", \"title\": \"BERTweet\", \"tokens\": 666, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"799e4bc8698b6093d431932732d68fe3939dd0c6de333b602a6583f27b85f140\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2009, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a17a6eff-44aa-4573-85c5-45d52c18f161", "doc_id": "a17a6eff-44aa-4573-85c5-45d52c18f161", "ref_doc_id": "a17a6eff-44aa-4573-85c5-45d52c18f161", "chroma:document": "# BERTweet\n\n## Overview\n\nThe BERTweet model was proposed in [BERTweet: A pre-trained language model for English Tweets](https://www.aclweb.org/anthology/2020.emnlp-demos.2.pdf) by Dat Quoc Nguyen, Thanh Vu, Anh Tuan Nguyen.\n\nThe abstract from the paper is the following:\n\n*We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having\nthe same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et\nal., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al.,\n2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks:\nPart-of-speech tagging, Named-entity recognition and text classification.*\n\nThis model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BERTweet).\n\n## Usage example\n\n```python\n>>> import torch\n>>> from transformers import AutoModel, AutoTokenizer\n\n>>> bertweet = AutoModel.from_pretrained(\"vinai/bertweet-base\")\n\n>>> # For transformers v4.x+:\n>>> tokenizer = AutoTokenizer.from_pretrained(\"vinai/bertweet-base\", use_fast=False)\n\n>>> # For transformers v3.x:\n>>> # tokenizer = AutoTokenizer.from_pretrained(\"vinai/bertweet-base\")\n\n>>> # INPUT TWEET IS ALREADY NORMALIZED!\n>>> line = \"SC has first two presumptive cases of coronavirus , DHEC confirms HTTPURL via @USER :cry:\"\n\n>>> input_ids = torch.tensor([tokenizer.encode(line)])\n\n>>> with torch.no_grad():\n...     features = bertweet(input_ids)  # Models outputs are now tuples\n\n>>> # With TensorFlow 2.0+:\n>>> # from transformers import TFAutoModel\n>>> # bertweet = TFAutoModel.from_pretrained(\"vinai/bertweet-base\")\n```\n\n<Tip> \n\nThis implementation is the same as BERT, except for tokenization method. Refer to [BERT documentation](bert) for \nAPI reference information.  \n\n</Tip>\n\n## BertweetTokenizer\n\n[[autodoc]] BertweetTokenizer"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 122c5f58-4417-445c-8430-c2727c9d9fbf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bridgetower/", "title": "BridgeTower", "tokens": 1905, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"122c5f58-4417-445c-8430-c2727c9d9fbf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b28f5b5-b78e-4778-b446-9142182253ca\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3429e92a6be30d00d9b9f43d1bad6731628a0eac027f0b48e423fc386477ef1b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6c1aea25-0f76-4fe8-a249-641a1ac1cc7d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ba1302c65f57690c858d4e6bdddd72a9d6381e9f5c06d1bc499de11740d9bf84\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3530, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "doc_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "ref_doc_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "chroma:document": "# BridgeTower\n\n## Overview\n\nThe BridgeTower model was proposed in [BridgeTower: Building Bridges Between Encoders in Vision-Language Representative Learning](https://arxiv.org/abs/2206.08657) by Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan. The goal of this model is to build a\nbridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder thus achieving remarkable performance on various downstream tasks with almost negligible additional performance and computational costs.\n\nThis paper has been accepted to the [AAAI'23](https://aaai.org/Conferences/AAAI-23/) conference. \n\nThe abstract from the paper is the following:\n\n*Vision-Language (VL) models with the TWO-TOWER architecture have dominated visual-language representation learning in recent years.\nCurrent VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder.\nBoth approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BRIDGETOWER, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the crossmodal encoder.\nThis enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BRIDGETOWER achieves state-of-the-art performance on various downstream vision-language tasks.\nIn particular, on the VQAv2 test-std set, BRIDGETOWER achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs.\nNotably, when further scaling the model, BRIDGETOWER achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/bridgetower_architecture%20.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> BridgeTower architecture. Taken from the <a href=\"https://arxiv.org/abs/2206.08657\">original paper.</a> </small>\n\nThis model was contributed by [Anahita Bhiwandiwalla](https://huggingface.co/anahita-b), [Tiep Le](https://huggingface.co/Tile) and [Shaoyen Tseng](https://huggingface.co/shaoyent). The original code can be found [here](https://github.com/microsoft/BridgeTower).\n\n## Usage tips and examples\n\nBridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers.\nThe goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder.\nIn principle, one can apply any visual, textual or cross-modal encoder in the proposed architecture.\n\nThe [`BridgeTowerProcessor`] wraps [`RobertaTokenizer`] and [`BridgeTowerImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run contrastive learning using [`BridgeTowerProcessor`] and [`BridgeTowerForContrastiveLearning`]."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6c1aea25-0f76-4fe8-a249-641a1ac1cc7d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bridgetower/", "title": "BridgeTower", "tokens": 1905, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6c1aea25-0f76-4fe8-a249-641a1ac1cc7d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b28f5b5-b78e-4778-b446-9142182253ca\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3429e92a6be30d00d9b9f43d1bad6731628a0eac027f0b48e423fc386477ef1b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"122c5f58-4417-445c-8430-c2727c9d9fbf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7df99c6cd67388f0bc4961002d3570938e73837e174337eb2a561b5c9e6bb0d3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cdd576da-5a9e-4897-89c6-45990336a346\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"638c9bcc3d6d1d32dae21daf4f2305f02b475b9777f959b10d977fb235f55080\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1792, \"end_char_idx\": 4456, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "doc_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "ref_doc_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "chroma:document": "In particular, on the VQAv2 test-std set, BRIDGETOWER achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs.\nNotably, when further scaling the model, BRIDGETOWER achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/bridgetower_architecture%20.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> BridgeTower architecture. Taken from the <a href=\"https://arxiv.org/abs/2206.08657\">original paper.</a> </small>\n\nThis model was contributed by [Anahita Bhiwandiwalla](https://huggingface.co/anahita-b), [Tiep Le](https://huggingface.co/Tile) and [Shaoyen Tseng](https://huggingface.co/shaoyent). The original code can be found [here](https://github.com/microsoft/BridgeTower).\n\n## Usage tips and examples\n\nBridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers.\nThe goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder.\nIn principle, one can apply any visual, textual or cross-modal encoder in the proposed architecture.\n\nThe [`BridgeTowerProcessor`] wraps [`RobertaTokenizer`] and [`BridgeTowerImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run contrastive learning using [`BridgeTowerProcessor`] and [`BridgeTowerForContrastiveLearning`].\n```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning\n>>> import requests\n>>> from PIL import Image\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [\"An image of two cats chilling on a couch\", \"A football player scoring a goal\"]\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-large-itm-mlm-itc\")\n>>> model = BridgeTowerForContrastiveLearning.from_pretrained(\"BridgeTower/bridgetower-large-itm-mlm-itc\")\n\n>>> # forward pass\n>>> scores = dict()\n>>> for text in texts:\n...     # prepare inputs\n...     encoding = processor(image, text, return_tensors=\"pt\")\n...     outputs = model(**encoding)\n...     scores[text] = outputs\n```\n\nThe following example shows how to run image-text retrieval using [`BridgeTowerProcessor`] and [`BridgeTowerForImageAndTextRetrieval`]."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cdd576da-5a9e-4897-89c6-45990336a346
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bridgetower/", "title": "BridgeTower", "tokens": 1905, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cdd576da-5a9e-4897-89c6-45990336a346\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b28f5b5-b78e-4778-b446-9142182253ca\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3429e92a6be30d00d9b9f43d1bad6731628a0eac027f0b48e423fc386477ef1b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6c1aea25-0f76-4fe8-a249-641a1ac1cc7d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1244d4ddf7d962a462e5f9b69cf46645ad8e07f4958adb2a9aaf1cb82c2e8106\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fe102d82-2be3-4c86-baa4-e169202d44dc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0144efbf4565b3a177d05da87af3406c60e3901d8d46b50eaed090d1bb3e7050\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2687, \"end_char_idx\": 5387, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "doc_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "ref_doc_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "chroma:document": "The original code can be found [here](https://github.com/microsoft/BridgeTower).\n\n## Usage tips and examples\n\nBridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers.\nThe goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder.\nIn principle, one can apply any visual, textual or cross-modal encoder in the proposed architecture.\n\nThe [`BridgeTowerProcessor`] wraps [`RobertaTokenizer`] and [`BridgeTowerImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run contrastive learning using [`BridgeTowerProcessor`] and [`BridgeTowerForContrastiveLearning`].\n```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning\n>>> import requests\n>>> from PIL import Image\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [\"An image of two cats chilling on a couch\", \"A football player scoring a goal\"]\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-large-itm-mlm-itc\")\n>>> model = BridgeTowerForContrastiveLearning.from_pretrained(\"BridgeTower/bridgetower-large-itm-mlm-itc\")\n\n>>> # forward pass\n>>> scores = dict()\n>>> for text in texts:\n...     # prepare inputs\n...     encoding = processor(image, text, return_tensors=\"pt\")\n...     outputs = model(**encoding)\n...     scores[text] = outputs\n```\n\nThe following example shows how to run image-text retrieval using [`BridgeTowerProcessor`] and [`BridgeTowerForImageAndTextRetrieval`].\n```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval\n>>> import requests\n>>> from PIL import Image\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [\"An image of two cats chilling on a couch\", \"A football player scoring a goal\"]\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n>>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n\n>>> # forward pass\n>>> scores = dict()\n>>> for text in texts:\n...     # prepare inputs\n...     encoding = processor(image, text, return_tensors=\"pt\")\n...     outputs = model(**encoding)\n...     scores[text] = outputs.logits[0, 1].item()\n```\n\nThe following example shows how to run masked language modeling using [`BridgeTowerProcessor`] and [`BridgeTowerForMaskedLM`]."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fe102d82-2be3-4c86-baa4-e169202d44dc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bridgetower/", "title": "BridgeTower", "tokens": 1905, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fe102d82-2be3-4c86-baa4-e169202d44dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b28f5b5-b78e-4778-b446-9142182253ca\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3429e92a6be30d00d9b9f43d1bad6731628a0eac027f0b48e423fc386477ef1b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cdd576da-5a9e-4897-89c6-45990336a346\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bridgetower/\", \"title\": \"BridgeTower\", \"tokens\": 1905, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c7267b75a7fc3f60b840a4db3c397223602864371f4c232c523eb467bd0ffeea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4457, \"end_char_idx\": 7476, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "doc_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "ref_doc_id": "8b28f5b5-b78e-4778-b446-9142182253ca", "chroma:document": "```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval\n>>> import requests\n>>> from PIL import Image\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [\"An image of two cats chilling on a couch\", \"A football player scoring a goal\"]\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n>>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n\n>>> # forward pass\n>>> scores = dict()\n>>> for text in texts:\n...     # prepare inputs\n...     encoding = processor(image, text, return_tensors=\"pt\")\n...     outputs = model(**encoding)\n...     scores[text] = outputs.logits[0, 1].item()\n```\n\nThe following example shows how to run masked language modeling using [`BridgeTowerProcessor`] and [`BridgeTowerForMaskedLM`].\n\n```python\n>>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM\n>>> from PIL import Image\n>>> import requests\n\n>>> url = \"http://images.cocodataset.org/val2017/000000360943.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw).convert(\"RGB\")\n>>> text = \"a <mask> looking out of the window\"\n\n>>> processor = BridgeTowerProcessor.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n>>> model = BridgeTowerForMaskedLM.from_pretrained(\"BridgeTower/bridgetower-base-itm-mlm\")\n\n>>> # prepare inputs\n>>> encoding = processor(image, text, return_tensors=\"pt\")\n\n>>> # forward pass\n>>> outputs = model(**encoding)\n\n>>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist())\n\n>>> print(results)\n.a cat looking out of the window.\n```\n\nTips:\n\n- This implementation of BridgeTower uses [`RobertaTokenizer`] to generate text embeddings and OpenAI's CLIP/ViT model to compute visual embeddings.\n- Checkpoints for pre-trained [bridgeTower-base](https://huggingface.co/BridgeTower/bridgetower-base) and [bridgetower masked language modeling and image text matching](https://huggingface.co/BridgeTower/bridgetower-base-itm-mlm) are released.\n- Please refer to [Table 5](https://arxiv.org/pdf/2206.08657.pdf) for BridgeTower's performance on Image Retrieval and other down stream tasks.\n- The PyTorch version of this model is only available in torch 1.10 and higher.\n\n\n## BridgeTowerConfig\n\n[[autodoc]] BridgeTowerConfig\n\n## BridgeTowerTextConfig\n\n[[autodoc]] BridgeTowerTextConfig\n\n## BridgeTowerVisionConfig\n\n[[autodoc]] BridgeTowerVisionConfig\n\n## BridgeTowerImageProcessor\n\n[[autodoc]] BridgeTowerImageProcessor\n    - preprocess\n\n## BridgeTowerProcessor\n\n[[autodoc]] BridgeTowerProcessor\n    - __call__\n\n## BridgeTowerModel\n\n[[autodoc]] BridgeTowerModel\n    - forward\n\n## BridgeTowerForContrastiveLearning\n\n[[autodoc]] BridgeTowerForContrastiveLearning\n    - forward\n\n## BridgeTowerForMaskedLM\n\n[[autodoc]] BridgeTowerForMaskedLM\n    - forward\n\n## BridgeTowerForImageAndTextRetrieval\n\n[[autodoc]] BridgeTowerForImageAndTextRetrieval\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4d8dda8b-d2ca-4e56-b833-18adaecceb6f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart/", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4d8dda8b-d2ca-4e56-b833-18adaecceb6f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85451877-c35d-4349-8466-33fdf153711b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a5fdb527c4138605027b398cfad83d429d9f99e5ebf8576eb146cbe8792d22e2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"96494f4a-e234-4816-ab11-158c49c5afbb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bc63cbd81fe2cf1a2a224adca8ec157427f4ec4c02ae8d8773874ef45f55c414\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3053, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85451877-c35d-4349-8466-33fdf153711b", "doc_id": "85451877-c35d-4349-8466-33fdf153711b", "ref_doc_id": "85451877-c35d-4349-8466-33fdf153711b", "chroma:document": "# BART\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=bart\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-bart-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/bart-large-mnli\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Bart model was proposed in [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation,\nTranslation, and Comprehension](https://arxiv.org/abs/1910.13461) by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan\nGhazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer on 29 Oct, 2019.\n\nAccording to the abstract,\n\n- Bart uses a standard seq2seq/machine translation architecture with a bidirectional encoder (like BERT) and a\n  left-to-right decoder (like GPT).\n- The pretraining task involves randomly shuffling the order of the original sentences and a novel in-filling scheme,\n  where spans of text are replaced with a single mask token.\n- BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It\n  matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new\n  state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains\n  of up to 6 ROUGE.\n\nThis model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/bart).\n\n## Usage tips:\n\n- BART is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- Sequence-to-sequence model with an encoder and a decoder. Encoder is fed a corrupted version of the tokens, decoder is fed the original tokens (but has a mask to hide the future words like a regular transformers decoder). A composition of the following transformations are applied on the pretraining tasks for the encoder:\n\n  * mask random tokens (like in BERT)\n  * delete random tokens\n  * mask a span of k tokens with a single mask token (a span of 0 tokens is an insertion of a mask token)\n  * permute sentences\n  * rotate the document to make it start at a specific token\n\n## Implementation Notes\n\n- Bart doesn't use `token_type_ids` for sequence classification. Use [`BartTokenizer`] or\n  [`~BartTokenizer.encode`] to get the proper splitting.\n- The forward pass of [`BartModel`] will create the `decoder_input_ids` if they are not passed.\n  This is different than some other modeling APIs. A typical use case of this feature is mask filling.\n- Model predictions are intended to be identical to the original implementation when\n  `forced_bos_token_id=0`. This only works, however, if the string you pass to\n  [`fairseq.encode`] starts with a space.\n- [`~generation.GenerationMixin.generate`] should be used for conditional generation tasks like\n  summarization, see the example in that docstrings."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 96494f4a-e234-4816-ab11-158c49c5afbb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart/", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"96494f4a-e234-4816-ab11-158c49c5afbb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85451877-c35d-4349-8466-33fdf153711b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a5fdb527c4138605027b398cfad83d429d9f99e5ebf8576eb146cbe8792d22e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4d8dda8b-d2ca-4e56-b833-18adaecceb6f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"56ab8689238d0673b1820f4eb3a2c39dc2e80e6fd0e5edfdb683e9e2965324ab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3cb10c69-456d-4449-ac56-8f8f66aee7f0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"08fdad219731f1b678ae7f752074bc1dc2135181b463433064b3ebb8cfb34f9d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1423, \"end_char_idx\": 4668, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85451877-c35d-4349-8466-33fdf153711b", "doc_id": "85451877-c35d-4349-8466-33fdf153711b", "ref_doc_id": "85451877-c35d-4349-8466-33fdf153711b", "chroma:document": "This model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/bart).\n\n## Usage tips:\n\n- BART is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- Sequence-to-sequence model with an encoder and a decoder. Encoder is fed a corrupted version of the tokens, decoder is fed the original tokens (but has a mask to hide the future words like a regular transformers decoder). A composition of the following transformations are applied on the pretraining tasks for the encoder:\n\n  * mask random tokens (like in BERT)\n  * delete random tokens\n  * mask a span of k tokens with a single mask token (a span of 0 tokens is an insertion of a mask token)\n  * permute sentences\n  * rotate the document to make it start at a specific token\n\n## Implementation Notes\n\n- Bart doesn't use `token_type_ids` for sequence classification. Use [`BartTokenizer`] or\n  [`~BartTokenizer.encode`] to get the proper splitting.\n- The forward pass of [`BartModel`] will create the `decoder_input_ids` if they are not passed.\n  This is different than some other modeling APIs. A typical use case of this feature is mask filling.\n- Model predictions are intended to be identical to the original implementation when\n  `forced_bos_token_id=0`. This only works, however, if the string you pass to\n  [`fairseq.encode`] starts with a space.\n- [`~generation.GenerationMixin.generate`] should be used for conditional generation tasks like\n  summarization, see the example in that docstrings.\n- Models that load the *facebook/bart-large-cnn* weights will not have a `mask_token_id`, or be able to perform\n  mask-filling tasks.\n\n## Mask Filling\n\nThe `facebook/bart-base` and `facebook/bart-large` checkpoints can be used to fill multi-token masks.\n\n```python\nfrom transformers import BartForConditionalGeneration, BartTokenizer\n\nmodel = BartForConditionalGeneration.from_pretrained(\"facebook/bart-large\", forced_bos_token_id=0)\ntok = BartTokenizer.from_pretrained(\"facebook/bart-large\")\nexample_english_phrase = \"UN Chief Says There Is No <mask> in Syria\"\nbatch = tok(example_english_phrase, return_tensors=\"pt\")\ngenerated_ids = model.generate(batch[\"input_ids\"])\nassert tok.batch_decode(generated_ids, skip_special_tokens=True) == [\n    \"UN Chief Says There Is No Plan to Stop Chemical Weapons in Syria\"\n]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BART. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"summarization\"/>\n\n- A blog post on [Distributed Training: Train BART/T5 for Summarization using \ud83e\udd17 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).\n- A notebook on how to [finetune BART for summarization with fastai using blurr](https://colab.research.google.com/github/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3cb10c69-456d-4449-ac56-8f8f66aee7f0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart/", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3cb10c69-456d-4449-ac56-8f8f66aee7f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85451877-c35d-4349-8466-33fdf153711b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a5fdb527c4138605027b398cfad83d429d9f99e5ebf8576eb146cbe8792d22e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"96494f4a-e234-4816-ab11-158c49c5afbb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d0ee713df18ec2ee7ee4de22f7f826d9d595bf22d8b072a24b37cd38a1a7eacf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"996de814-6074-40ba-8d7b-1f181e84d051\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4e3fe4e5cc1790648957979804f2ae6d2193fc2611b17f2b4ce141d9a077e52d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3054, \"end_char_idx\": 5969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85451877-c35d-4349-8466-33fdf153711b", "doc_id": "85451877-c35d-4349-8466-33fdf153711b", "ref_doc_id": "85451877-c35d-4349-8466-33fdf153711b", "chroma:document": "- Models that load the *facebook/bart-large-cnn* weights will not have a `mask_token_id`, or be able to perform\n  mask-filling tasks.\n\n## Mask Filling\n\nThe `facebook/bart-base` and `facebook/bart-large` checkpoints can be used to fill multi-token masks.\n\n```python\nfrom transformers import BartForConditionalGeneration, BartTokenizer\n\nmodel = BartForConditionalGeneration.from_pretrained(\"facebook/bart-large\", forced_bos_token_id=0)\ntok = BartTokenizer.from_pretrained(\"facebook/bart-large\")\nexample_english_phrase = \"UN Chief Says There Is No <mask> in Syria\"\nbatch = tok(example_english_phrase, return_tensors=\"pt\")\ngenerated_ids = model.generate(batch[\"input_ids\"])\nassert tok.batch_decode(generated_ids, skip_special_tokens=True) == [\n    \"UN Chief Says There Is No Plan to Stop Chemical Weapons in Syria\"\n]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BART. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"summarization\"/>\n\n- A blog post on [Distributed Training: Train BART/T5 for Summarization using \ud83e\udd17 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).\n- A notebook on how to [finetune BART for summarization with fastai using blurr](https://colab.research.google.com/github/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune BART for summarization in two languages with Trainer class](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n- [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).\n- An example of how to train [`BartForConditionalGeneration`] with a Hugging Face `datasets` object can be found in this [forum discussion](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904)\n- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the \ud83e\udd17 Hugging Face course."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 996de814-6074-40ba-8d7b-1f181e84d051
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart/", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"996de814-6074-40ba-8d7b-1f181e84d051\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85451877-c35d-4349-8466-33fdf153711b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a5fdb527c4138605027b398cfad83d429d9f99e5ebf8576eb146cbe8792d22e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3cb10c69-456d-4449-ac56-8f8f66aee7f0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b74ab73c0a52e12a0a0867bef85da1e1f9ad3a22282a0f1c4c2c461c93d3bd4e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"570fd572-316b-454e-9397-df3621c3f4e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"72d14d2adb844c7a006c097fa518901f4a136517c440a5e71713d5e0774293c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4669, \"end_char_idx\": 7741, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85451877-c35d-4349-8466-33fdf153711b", "doc_id": "85451877-c35d-4349-8466-33fdf153711b", "ref_doc_id": "85451877-c35d-4349-8466-33fdf153711b", "chroma:document": "\ud83c\udf0e\n- A notebook on how to [finetune BART for summarization in two languages with Trainer class](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n- [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).\n- An example of how to train [`BartForConditionalGeneration`] with a Hugging Face `datasets` object can be found in this [forum discussion](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904)\n- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the \ud83e\udd17 Hugging Face course.\n- [Summarization task guide](../tasks/summarization)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"translation\"/>\n\n- A notebook on how to [finetune mBART using Seq2SeqTrainer for Hindi to English translation](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 570fd572-316b-454e-9397-df3621c3f4e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart/", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"570fd572-316b-454e-9397-df3621c3f4e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85451877-c35d-4349-8466-33fdf153711b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a5fdb527c4138605027b398cfad83d429d9f99e5ebf8576eb146cbe8792d22e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"996de814-6074-40ba-8d7b-1f181e84d051\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"26208c77b6f3bc9495d885bbb0abbc42c53691660bcde34a8359fddcdb640630\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"01c8aa81-3dfd-4582-9e56-22689c303417\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6ce315a056bad78fd4f1cfc6ddc01aa745fa2f7738a06635767d60bbb3b534ff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6398, \"end_char_idx\": 8590, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85451877-c35d-4349-8466-33fdf153711b", "doc_id": "85451877-c35d-4349-8466-33fdf153711b", "ref_doc_id": "85451877-c35d-4349-8466-33fdf153711b", "chroma:document": "- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"translation\"/>\n\n- A notebook on how to [finetune mBART using Seq2SeqTrainer for Hindi to English translation](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n- [Translation task guide](../tasks/translation)\n\nSee also:\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Distilled checkpoints](https://huggingface.co/models?search=distilbart) are described in this [paper](https://arxiv.org/abs/2010.13002).\n\n## BartConfig\n\n[[autodoc]] BartConfig\n    - all\n\n## BartTokenizer\n\n[[autodoc]] BartTokenizer\n    - all\n\n## BartTokenizerFast\n\n[[autodoc]] BartTokenizerFast\n    - all"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 01c8aa81-3dfd-4582-9e56-22689c303417
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bart/", "title": "BART", "tokens": 2635, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"01c8aa81-3dfd-4582-9e56-22689c303417\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85451877-c35d-4349-8466-33fdf153711b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a5fdb527c4138605027b398cfad83d429d9f99e5ebf8576eb146cbe8792d22e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"570fd572-316b-454e-9397-df3621c3f4e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bart/\", \"title\": \"BART\", \"tokens\": 2635, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e025897d0d71b4210d45e837177f8edf2f2d6f4b5efef58144122baf9fa33284\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7026, \"end_char_idx\": 9801, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85451877-c35d-4349-8466-33fdf153711b", "doc_id": "85451877-c35d-4349-8466-33fdf153711b", "ref_doc_id": "85451877-c35d-4349-8466-33fdf153711b", "chroma:document": "- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"translation\"/>\n\n- A notebook on how to [finetune mBART using Seq2SeqTrainer for Hindi to English translation](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb). \ud83c\udf0e\n- [`BartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).\n- [`TFBartForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n- [Translation task guide](../tasks/translation)\n\nSee also:\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Distilled checkpoints](https://huggingface.co/models?search=distilbart) are described in this [paper](https://arxiv.org/abs/2010.13002).\n\n## BartConfig\n\n[[autodoc]] BartConfig\n    - all\n\n## BartTokenizer\n\n[[autodoc]] BartTokenizer\n    - all\n\n## BartTokenizerFast\n\n[[autodoc]] BartTokenizerFast\n    - all\n\n\n<frameworkcontent>\n<pt>\n\n## BartModel\n\n[[autodoc]] BartModel\n    - forward\n\n## BartForConditionalGeneration\n\n[[autodoc]] BartForConditionalGeneration\n    - forward\n\n## BartForSequenceClassification\n\n[[autodoc]] BartForSequenceClassification\n    - forward\n\n## BartForQuestionAnswering\n\n[[autodoc]] BartForQuestionAnswering\n    - forward\n\n## BartForCausalLM\n\n[[autodoc]] BartForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFBartModel\n\n[[autodoc]] TFBartModel\n    - call\n\n## TFBartForConditionalGeneration\n\n[[autodoc]] TFBartForConditionalGeneration\n    - call\n\n## TFBartForSequenceClassification\n\n[[autodoc]] TFBartForSequenceClassification\n    - call\n\n</tf>\n<jax>\n\n## FlaxBartModel\n\n[[autodoc]] FlaxBartModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxBartForConditionalGeneration\n\n[[autodoc]] FlaxBartForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n## FlaxBartForSequenceClassification\n\n[[autodoc]] FlaxBartForSequenceClassification\n    - __call__\n    - encode\n    - decode\n\n## FlaxBartForQuestionAnswering\n\n[[autodoc]] FlaxBartForQuestionAnswering\n    - __call__\n    - encode\n    - decode\n\n## FlaxBartForCausalLM\n\n[[autodoc]] FlaxBartForCausalLM\n    - __call__\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 29200d15-9e36-491d-add1-76a7ce045deb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapex/", "title": "TAPEX", "tokens": 1845, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"29200d15-9e36-491d-add1-76a7ce045deb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"78b6fecc-7ebe-476c-a04a-13d2f53054c8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"715ee0c5dfb9fd881b1b7a8a2af570ff085239c27379b116c4a75df9f7d8aee1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d4fc51da-b62e-4504-978d-6ea9edb723d1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"62f702a46329fc7552c2e9f3f0260f779d060428531b8161eeee188e28dfcad5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3054, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "doc_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "ref_doc_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "chroma:document": "# TAPEX\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe TAPEX model was proposed in [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://arxiv.org/abs/2107.07653) by Qian Liu,\nBei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, Jian-Guang Lou. TAPEX pre-trains a BART model to solve synthetic SQL queries, after\nwhich it can be fine-tuned to answer natural language questions related to tabular data, as well as performing table fact checking. \n\nTAPEX has been fine-tuned on several datasets: \n- [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) (Sequential Question Answering by Microsoft)\n- [WTQ](https://github.com/ppasupat/WikiTableQuestions) (Wiki Table Questions by Stanford University)\n- [WikiSQL](https://github.com/salesforce/WikiSQL) (by Salesforce)\n- [TabFact](https://tabfact.github.io/) (by USCB NLP Lab).\n\nThe abstract from the paper is the following:\n\n*Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is\nstill a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we\npropose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically\nsynthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL\nexecutor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that\nTAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes improvements\non the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy\nto 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs\nand to achieve new state-of-the-art results on various downstream tasks.*\n\n## Usage tips\n\n- TAPEX is a generative (seq2seq) model. One can directly plug in the weights of TAPEX into a BART model. \n- TAPEX has checkpoints on the hub that are either pre-trained only, or fine-tuned on WTQ, SQA, WikiSQL and TabFact.\n- Sentences + tables are presented to the model as `sentence + \" \" + linearized table`. The linearized table has the following format: \n  `col: col1 | col2 | col 3 row 1 : val1 | val2 | val3 row 2 : ...`.\n- TAPEX has its own tokenizer, that allows to prepare all data for the model easily."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d4fc51da-b62e-4504-978d-6ea9edb723d1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapex/", "title": "TAPEX", "tokens": 1845, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d4fc51da-b62e-4504-978d-6ea9edb723d1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"78b6fecc-7ebe-476c-a04a-13d2f53054c8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"715ee0c5dfb9fd881b1b7a8a2af570ff085239c27379b116c4a75df9f7d8aee1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"29200d15-9e36-491d-add1-76a7ce045deb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e1b926c126832f2166f3394adf356e2ab427bead67ba5c918877ad8dad6ad94e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fec4f6fe-6ae2-4177-b1bd-c10e0197bf95\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fb1700e4257e50b62c49135b7c0b8e5482f2119140d9f520c0c243654b4d8d97\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1480, \"end_char_idx\": 4528, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "doc_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "ref_doc_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "chroma:document": "In this paper, we\npropose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically\nsynthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL\nexecutor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that\nTAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes improvements\non the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy\nto 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs\nand to achieve new state-of-the-art results on various downstream tasks.*\n\n## Usage tips\n\n- TAPEX is a generative (seq2seq) model. One can directly plug in the weights of TAPEX into a BART model. \n- TAPEX has checkpoints on the hub that are either pre-trained only, or fine-tuned on WTQ, SQA, WikiSQL and TabFact.\n- Sentences + tables are presented to the model as `sentence + \" \" + linearized table`. The linearized table has the following format: \n  `col: col1 | col2 | col 3 row 1 : val1 | val2 | val3 row 2 : ...`.\n- TAPEX has its own tokenizer, that allows to prepare all data for the model easily. One can pass Pandas DataFrames and strings to the tokenizer,\n  and it will automatically create the `input_ids` and `attention_mask` (as shown in the usage examples below). \n\n### Usage: inference\n\nBelow, we illustrate how to use TAPEX for table question answering. As one can see, one can directly plug in the weights of TAPEX into a BART model.\nWe use the [Auto API](auto), which will automatically instantiate the appropriate tokenizer ([`TapexTokenizer`]) and model ([`BartForConditionalGeneration`]) for us,\nbased on the configuration file of the checkpoint on the hub.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import pandas as pd\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/tapex-large-finetuned-wtq\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"microsoft/tapex-large-finetuned-wtq\")\n\n>>> # prepare table + question\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> question = \"how many movies does Leonardo Di Caprio have?\"\n\n>>> encoding = tokenizer(table, question, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> predicted_answer = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> print(predicted_answer)\n53\n```\n\nNote that [`TapexTokenizer`] also supports batched inference."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fec4f6fe-6ae2-4177-b1bd-c10e0197bf95
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapex/", "title": "TAPEX", "tokens": 1845, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fec4f6fe-6ae2-4177-b1bd-c10e0197bf95\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"78b6fecc-7ebe-476c-a04a-13d2f53054c8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"715ee0c5dfb9fd881b1b7a8a2af570ff085239c27379b116c4a75df9f7d8aee1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d4fc51da-b62e-4504-978d-6ea9edb723d1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"88be7f79328ec115b2c00b6f241c34539fd47993489b9f32bddda01237ce4ca7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"abd6f0f9-cad6-4a0f-aa58-0b168ec974ca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"adcbf080a422dcce3158cd1d53c42609d741a86706bf00096f88329952cc07d1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2970, \"end_char_idx\": 5817, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "doc_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "ref_doc_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "chroma:document": "- TAPEX has its own tokenizer, that allows to prepare all data for the model easily. One can pass Pandas DataFrames and strings to the tokenizer,\n  and it will automatically create the `input_ids` and `attention_mask` (as shown in the usage examples below). \n\n### Usage: inference\n\nBelow, we illustrate how to use TAPEX for table question answering. As one can see, one can directly plug in the weights of TAPEX into a BART model.\nWe use the [Auto API](auto), which will automatically instantiate the appropriate tokenizer ([`TapexTokenizer`]) and model ([`BartForConditionalGeneration`]) for us,\nbased on the configuration file of the checkpoint on the hub.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import pandas as pd\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/tapex-large-finetuned-wtq\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"microsoft/tapex-large-finetuned-wtq\")\n\n>>> # prepare table + question\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> question = \"how many movies does Leonardo Di Caprio have?\"\n\n>>> encoding = tokenizer(table, question, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> predicted_answer = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> print(predicted_answer)\n53\n```\n\nNote that [`TapexTokenizer`] also supports batched inference. Hence, one can provide a batch of different tables/questions, or a batch of a single table\nand multiple questions, or a batch of a single query and multiple tables. Let's illustrate this:\n\n```python\n>>> # prepare table + question\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> questions = [\n...     \"how many movies does Leonardo Di Caprio have?\",\n...     \"which actor has 69 movies?\",\n...     \"what's the first name of the actor who has 87 movies?\",\n... ]\n>>> encoding = tokenizer(table, questions, padding=True, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n[' 53', ' george clooney', ' brad pitt']\n```\n\nIn case one wants to do table verification (i.e. the task of determining whether a given sentence is supported or refuted by the contents\nof a table), one can instantiate a [`BartForSequenceClassification`] model. TAPEX has checkpoints on the hub fine-tuned on TabFact, an important\nbenchmark for table fact checking (it achieves 84% accuracy). The code example below again leverages the [Auto API](auto)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: abd6f0f9-cad6-4a0f-aa58-0b168ec974ca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapex/", "title": "TAPEX", "tokens": 1845, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"abd6f0f9-cad6-4a0f-aa58-0b168ec974ca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"78b6fecc-7ebe-476c-a04a-13d2f53054c8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"715ee0c5dfb9fd881b1b7a8a2af570ff085239c27379b116c4a75df9f7d8aee1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fec4f6fe-6ae2-4177-b1bd-c10e0197bf95\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex/\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"08eb6b2d60cb6cfe71620fb3482b07265bc70de995410cc96ceaad7c90ec08cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4159, \"end_char_idx\": 6924, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "doc_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "ref_doc_id": "78b6fecc-7ebe-476c-a04a-13d2f53054c8", "chroma:document": ">>> encoding = tokenizer(table, question, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> predicted_answer = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> print(predicted_answer)\n53\n```\n\nNote that [`TapexTokenizer`] also supports batched inference. Hence, one can provide a batch of different tables/questions, or a batch of a single table\nand multiple questions, or a batch of a single query and multiple tables. Let's illustrate this:\n\n```python\n>>> # prepare table + question\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> questions = [\n...     \"how many movies does Leonardo Di Caprio have?\",\n...     \"which actor has 69 movies?\",\n...     \"what's the first name of the actor who has 87 movies?\",\n... ]\n>>> encoding = tokenizer(table, questions, padding=True, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n[' 53', ' george clooney', ' brad pitt']\n```\n\nIn case one wants to do table verification (i.e. the task of determining whether a given sentence is supported or refuted by the contents\nof a table), one can instantiate a [`BartForSequenceClassification`] model. TAPEX has checkpoints on the hub fine-tuned on TabFact, an important\nbenchmark for table fact checking (it achieves 84% accuracy). The code example below again leverages the [Auto API](auto).\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/tapex-large-finetuned-tabfact\")\n>>> model = AutoModelForSequenceClassification.from_pretrained(\"microsoft/tapex-large-finetuned-tabfact\")\n\n>>> # prepare table + sentence\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> sentence = \"George Clooney has 30 movies\"\n\n>>> encoding = tokenizer(table, sentence, return_tensors=\"pt\")\n\n>>> # forward pass\n>>> outputs = model(**encoding)\n\n>>> # print prediction\n>>> predicted_class_idx = outputs.logits[0].argmax(dim=0).item()\n>>> print(model.config.id2label[predicted_class_idx])\nRefused\n```\n\n<Tip> \n\nTAPEX architecture is the same as BART, except for tokenization. Refer to [BART documentation](bart) for information on \nconfiguration classes and their parameters. TAPEX-specific tokenizer is documented below.  \n\n</Tip>\n\n## TapexTokenizer\n\n[[autodoc]] TapexTokenizer\n    - __call__\n    - save_vocabulary"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0bd2f628-b507-48c9-bbaa-4f53355b4910
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/efficientformer/", "title": "EfficientFormer", "tokens": 1010, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0bd2f628-b507-48c9-bbaa-4f53355b4910\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer/\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4fc0ef81-e3db-4275-8b1b-74e030132265\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer/\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f5492adbbce999a4de114cdd9fe40fb58a2bad1c07617fb45f601a2b3629675a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"00be89fe-09d1-4786-8374-c2807dd22c43\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ec14cfc786519fc9b478c6c1180b2a63cfb0a34f01a147f1a522728e6af675e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2811, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4fc0ef81-e3db-4275-8b1b-74e030132265", "doc_id": "4fc0ef81-e3db-4275-8b1b-74e030132265", "ref_doc_id": "4fc0ef81-e3db-4275-8b1b-74e030132265", "chroma:document": "# EfficientFormer\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe EfficientFormer model was proposed in [EfficientFormer: Vision Transformers at MobileNet Speed](https://arxiv.org/abs/2206.01191)\nby Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, Jian Ren.  EfficientFormer proposes a\ndimension-consistent pure transformer that can be run on mobile devices for dense prediction tasks like image classification, object\ndetection and semantic segmentation.\n\nThe abstract from the paper is the following:\n\n*Vision Transformers (ViT) have shown rapid progress in computer vision tasks, achieving promising results on various benchmarks.\nHowever, due to the massive number of parameters and model design, e.g., attention mechanism, ViT-based models are generally\ntimes slower than lightweight convolutional networks. Therefore, the deployment of ViT for real-time applications is particularly\nchallenging, especially on resource-constrained hardware such as mobile devices. Recent efforts try to reduce the computation\ncomplexity of ViT through network architecture search or hybrid design with MobileNet block, yet the inference speed is still\nunsatisfactory. This leads to an important question: can transformers run as fast as MobileNet while obtaining high performance?\nTo answer this, we first revisit the network architecture and operators used in ViT-based models and identify inefficient designs.\nThen we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm.\nFinally, we perform latency-driven slimming to get a series of final models dubbed EfficientFormer.\nExtensive experiments show the superiority of EfficientFormer in performance and speed on mobile devices.\nOur fastest model, EfficientFormer-L1, achieves 79.2% top-1 accuracy on ImageNet-1K with only 1.6 ms inference latency on\niPhone 12 (compiled with CoreML), which { runs as fast as MobileNetV2\u00d71.4 (1.6 ms, 74.7% top-1),} and our largest model,\nEfficientFormer-L7, obtains 83.3% accuracy with only 7.0 ms latency. Our work proves that properly designed transformers can\nreach extremely low latency on mobile devices while maintaining high performance.*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03) and [Bearnardd](https://huggingface.co/Bearnardd).\nThe original code can be found [here](https://github.com/snap-research/EfficientFormer). The TensorFlow version of this model was added by [D-Roberts](https://huggingface.co/D-Roberts)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 00be89fe-09d1-4786-8374-c2807dd22c43
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/efficientformer/", "title": "EfficientFormer", "tokens": 1010, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"00be89fe-09d1-4786-8374-c2807dd22c43\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer/\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4fc0ef81-e3db-4275-8b1b-74e030132265\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer/\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f5492adbbce999a4de114cdd9fe40fb58a2bad1c07617fb45f601a2b3629675a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0bd2f628-b507-48c9-bbaa-4f53355b4910\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer/\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"838c36eb5dcb1c634d5eea39d72c5fb61465a5be9739e886e31930b6086e1621\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1148, \"end_char_idx\": 3747, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4fc0ef81-e3db-4275-8b1b-74e030132265", "doc_id": "4fc0ef81-e3db-4275-8b1b-74e030132265", "ref_doc_id": "4fc0ef81-e3db-4275-8b1b-74e030132265", "chroma:document": "Therefore, the deployment of ViT for real-time applications is particularly\nchallenging, especially on resource-constrained hardware such as mobile devices. Recent efforts try to reduce the computation\ncomplexity of ViT through network architecture search or hybrid design with MobileNet block, yet the inference speed is still\nunsatisfactory. This leads to an important question: can transformers run as fast as MobileNet while obtaining high performance?\nTo answer this, we first revisit the network architecture and operators used in ViT-based models and identify inefficient designs.\nThen we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm.\nFinally, we perform latency-driven slimming to get a series of final models dubbed EfficientFormer.\nExtensive experiments show the superiority of EfficientFormer in performance and speed on mobile devices.\nOur fastest model, EfficientFormer-L1, achieves 79.2% top-1 accuracy on ImageNet-1K with only 1.6 ms inference latency on\niPhone 12 (compiled with CoreML), which { runs as fast as MobileNetV2\u00d71.4 (1.6 ms, 74.7% top-1),} and our largest model,\nEfficientFormer-L7, obtains 83.3% accuracy with only 7.0 ms latency. Our work proves that properly designed transformers can\nreach extremely low latency on mobile devices while maintaining high performance.*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03) and [Bearnardd](https://huggingface.co/Bearnardd).\nThe original code can be found [here](https://github.com/snap-research/EfficientFormer). The TensorFlow version of this model was added by [D-Roberts](https://huggingface.co/D-Roberts).\n\n## Documentation resources\n\n- [Image classification task guide](../tasks/image_classification)\n\n## EfficientFormerConfig\n\n[[autodoc]] EfficientFormerConfig\n\n## EfficientFormerImageProcessor\n\n[[autodoc]] EfficientFormerImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## EfficientFormerModel\n\n[[autodoc]] EfficientFormerModel\n    - forward\n\n## EfficientFormerForImageClassification\n\n[[autodoc]] EfficientFormerForImageClassification\n    - forward\n\n## EfficientFormerForImageClassificationWithTeacher\n\n[[autodoc]] EfficientFormerForImageClassificationWithTeacher\n    - forward\n\n</pt>\n<tf>\n\n## TFEfficientFormerModel\n\n[[autodoc]] TFEfficientFormerModel\n    - call\n\n## TFEfficientFormerForImageClassification\n\n[[autodoc]] TFEfficientFormerForImageClassification\n    - call\n\n## TFEfficientFormerForImageClassificationWithTeacher\n\n[[autodoc]] TFEfficientFormerForImageClassificationWithTeacher\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2593410c-5c89-4ac2-9151-fbda0641f103
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/madlad-400/", "title": "MADLAD-400", "tokens": 797, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2593410c-5c89-4ac2-9151-fbda0641f103\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/madlad-400/\", \"title\": \"MADLAD-400\", \"tokens\": 797, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"933194e0-95dd-480e-ba3c-d92860550eb6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/madlad-400/\", \"title\": \"MADLAD-400\", \"tokens\": 797, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8775e3336db27e619f03d67b1848837e0dd11696f1971ac4c1516bba452e0341\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2495, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "933194e0-95dd-480e-ba3c-d92860550eb6", "doc_id": "933194e0-95dd-480e-ba3c-d92860550eb6", "ref_doc_id": "933194e0-95dd-480e-ba3c-d92860550eb6", "chroma:document": "# MADLAD-400\n\n## Overview\n\nMADLAD-400 models were released in the paper [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](MADLAD-400: A Multilingual And Document-Level Large Audited Dataset). \n\nThe abstract from the paper is the following: \n\n*We introduce MADLAD-400, a manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss \nthe limitations revealed by self-auditing MADLAD-400, and the role data auditing\nhad in the dataset creation process. We then train and release a 10.7B-parameter\nmultilingual machine translation model on 250 billion tokens covering over 450\nlanguages using publicly available data, and find that it is competitive with models\nthat are significantly larger, and report the results on different domains. In addition, we train a 8B-parameter language model, and assess the results on few-shot\ntranslation. We make the baseline models 1\navailable to the research community.*\n\nThis model was added by [Juarez Bochi](https://huggingface.co/jbochi). The original checkpoints can be found [here](https://github.com/google-research/google-research/tree/master/madlad_400). \n\nThis is a machine translation model that supports many low-resource languages, and that is competitive with models that are significantly larger.\n\nOne can directly use MADLAD-400 weights without finetuning the model:\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/madlad400-3b-mt\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/madlad400-3b-mt\")\n\n>>> inputs = tokenizer(\"<2pt> I love pizza!\", return_tensors=\"pt\")\n>>> outputs = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['Eu amo pizza!']\n```\n\nGoogle has released the following variants:\n\n- [google/madlad400-3b-mt](https://huggingface.co/google/madlad400-3b-mt)\n\n- [google/madlad400-7b-mt](https://huggingface.co/google/madlad400-7b-mt)\n\n- [google/madlad400-7b-mt-bt](https://huggingface.co/google/madlad400-7b-mt-bt)\n\n- [google/madlad400-10b-mt](https://huggingface.co/google/madlad400-10b-mt)\n\nThe original checkpoints can be found [here](https://github.com/google-research/google-research/tree/master/madlad_400).\n\n<Tip>\n\nRefer to [T5's documentation page](t5) for all API references, code examples, and notebooks. For more details regarding training and evaluation of the MADLAD-400, refer to the model card.\n\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 659f0742-0361-4ead-b0ea-6dd0f8441298
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mamba/", "title": "Mamba", "tokens": 1262, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"659f0742-0361-4ead-b0ea-6dd0f8441298\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba/\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"accdfa47-9f96-498f-ac12-0ab2f32deb33\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba/\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4005dd86feaf054525f10e66d318f60b95197fe983e066b9bec166cd56836ee6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a600e51b-ba7e-4bd9-ad36-8cc57fa3b10b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"95b10197a6a934ae901fe83bf7f516c171a5e8b9656efa9e72d8b2843af7c9f4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3508, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "accdfa47-9f96-498f-ac12-0ab2f32deb33", "doc_id": "accdfa47-9f96-498f-ac12-0ab2f32deb33", "ref_doc_id": "accdfa47-9f96-498f-ac12-0ab2f32deb33", "chroma:document": "# Mamba\n\n## Overview\n\nThe Mamba model was proposed in [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752) by Albert Gu and Tri Dao.\n\nThis model is a new paradigm architecture based on `state-space-models`. You can read more about the intuition behind these [here](https://srush.github.io/annotated-s4/).\n\nThe abstract from the paper is the following:\n\n*Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5\u00d7 higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.*\n\nTips:\n\n- Mamba is a new `state space model` architecture that rivals the classic Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of [FlashAttention](https://github.com/Dao-AILab/flash-attention).\n- Mamba stacks `mixer` layers, which are the equivalent of `Attention` layers. The core logic of `mamba` is held in the `MambaMixer` class.\n- Two implementations cohabit: one is optimized and uses fast cuda kernels, while the other one is naive but can run on any device!\n- The current implementation leverages the original cuda kernels: the equivalent of flash attention for Mamba are hosted in the [`mamba-ssm`](https://github.com/state-spaces/mamba) and the [`causal_conv1d`](https://github.com/Dao-AILab/causal-conv1d) repositories. Make sure to install them if your hardware supports them!\n- Contributions to make the naive path faster are welcome \ud83e\udd17\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/state-spaces/mamba).\n\n# Usage\n\n### A simple generation example: \n```python \nfrom transformers import MambaConfig, MambaForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"state-spaces/mamba-130m-hf\")\nmodel = MambaForCausalLM.from_pretrained(\"state-spaces/mamba-130m-hf\")\ninput_ids = tokenizer(\"Hey how are you doing?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a600e51b-ba7e-4bd9-ad36-8cc57fa3b10b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mamba/", "title": "Mamba", "tokens": 1262, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a600e51b-ba7e-4bd9-ad36-8cc57fa3b10b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba/\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"accdfa47-9f96-498f-ac12-0ab2f32deb33\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba/\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4005dd86feaf054525f10e66d318f60b95197fe983e066b9bec166cd56836ee6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"659f0742-0361-4ead-b0ea-6dd0f8441298\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba/\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a9c58af4ea752e726ce3c70b90aa6a0302a179b07b3814a3a5cee77846526c0c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1896, \"end_char_idx\": 4850, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "accdfa47-9f96-498f-ac12-0ab2f32deb33", "doc_id": "accdfa47-9f96-498f-ac12-0ab2f32deb33", "ref_doc_id": "accdfa47-9f96-498f-ac12-0ab2f32deb33", "chroma:document": "On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.*\n\nTips:\n\n- Mamba is a new `state space model` architecture that rivals the classic Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of [FlashAttention](https://github.com/Dao-AILab/flash-attention).\n- Mamba stacks `mixer` layers, which are the equivalent of `Attention` layers. The core logic of `mamba` is held in the `MambaMixer` class.\n- Two implementations cohabit: one is optimized and uses fast cuda kernels, while the other one is naive but can run on any device!\n- The current implementation leverages the original cuda kernels: the equivalent of flash attention for Mamba are hosted in the [`mamba-ssm`](https://github.com/state-spaces/mamba) and the [`causal_conv1d`](https://github.com/Dao-AILab/causal-conv1d) repositories. Make sure to install them if your hardware supports them!\n- Contributions to make the naive path faster are welcome \ud83e\udd17\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/state-spaces/mamba).\n\n# Usage\n\n### A simple generation example: \n```python \nfrom transformers import MambaConfig, MambaForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"state-spaces/mamba-130m-hf\")\nmodel = MambaForCausalLM.from_pretrained(\"state-spaces/mamba-130m-hf\")\ninput_ids = tokenizer(\"Hey how are you doing?\", return_tensors= \"pt\")[\"input_ids\"]\n\nout = model.generate(input_ids, max_new_tokens=10)\nprint(tokenizer.batch_decode(out))\n```\n\n### Peft finetuning\nThe slow version is not very stable for training, and the fast one needs `float32`!\n\n```python \nfrom datasets import load_dataset\nfrom trl import SFTTrainer\nfrom peft import LoraConfig\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments\nmodel_id = \"state-spaces/mamba-130m-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\ndataset = load_dataset(\"Abirate/english_quotes\", split=\"train\")\ntraining_args = TrainingArguments(\n    output_dir=\"./results\",\n    num_train_epochs=3,\n    per_device_train_batch_size=4,\n    logging_dir='./logs',\n    logging_steps=10,\n    learning_rate=2e-3\n)\nlora_config =  LoraConfig(\n        r=8,\n        target_modules=[\"x_proj\", \"embeddings\", \"in_proj\", \"out_proj\"],\n        task_type=\"CAUSAL_LM\",\n        bias=\"none\"\n)\ntrainer = SFTTrainer(\n    model=model,\n    tokenizer=tokenizer,\n    args=training_args,\n    peft_config=lora_config,\n    train_dataset=dataset,\n    dataset_text_field=\"quote\",\n)\ntrainer.train()\n```\n\n## MambaConfig\n\n[[autodoc]] MambaConfig\n\n## MambaModel\n\n[[autodoc]] MambaModel\n    - forward\n\n## MambaLMHeadModel\n\n[[autodoc]] MambaForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0920f873-0f23-4dfb-9039-8404d4a83448
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cvt/", "title": "Convolutional Vision Transformer (CvT)", "tokens": 1132, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0920f873-0f23-4dfb-9039-8404d4a83448\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt/\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2bb36a86-6fc3-4d31-a614-dad738a7899a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt/\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"debe40d862f949ea813b20aa13220f48fec090ee3a11d3768a7af494dd97c320\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"63698ae3-cba3-405c-876b-52b0fda7f009\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"74216f69059be64d5a69b28258bc72bb5fb25be6b43e90b714a897fcc53a4415\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3034, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2bb36a86-6fc3-4d31-a614-dad738a7899a", "doc_id": "2bb36a86-6fc3-4d31-a614-dad738a7899a", "ref_doc_id": "2bb36a86-6fc3-4d31-a614-dad738a7899a", "chroma:document": "# Convolutional Vision Transformer (CvT)\n\n## Overview\n\nThe CvT model was proposed in [CvT: Introducing Convolutions to Vision Transformers](https://arxiv.org/abs/2103.15808) by Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan and Lei Zhang. The Convolutional vision Transformer (CvT) improves the [Vision Transformer (ViT)](vit) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs.\n\nThe abstract from the paper is the following:\n\n*We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) \nin performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through \ntwo primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer \nblock leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) \nto the ViT architecture (\\ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (\\ie dynamic attention, \nglobal context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves \nstate-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, \nperformance gains are maintained when pretrained on larger datasets (\\eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on \nImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, \na crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks.*\n\nThis model was contributed by [anugunj](https://huggingface.co/anugunj). The original code can be found [here](https://github.com/microsoft/CvT).\n\n## Usage tips\n\n- CvT models are regular Vision Transformers, but trained with convolutions. They outperform the [original model (ViT)](vit) when fine-tuned on ImageNet-1K and CIFAR-100.\n- You can check out demo notebooks regarding inference as well as fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace [`ViTFeatureExtractor`] by [`AutoImageProcessor`] and [`ViTForImageClassification`] by [`CvtForImageClassification`]).\n- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of 14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CvT."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 63698ae3-cba3-405c-876b-52b0fda7f009
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cvt/", "title": "Convolutional Vision Transformer (CvT)", "tokens": 1132, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"63698ae3-cba3-405c-876b-52b0fda7f009\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt/\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2bb36a86-6fc3-4d31-a614-dad738a7899a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt/\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"debe40d862f949ea813b20aa13220f48fec090ee3a11d3768a7af494dd97c320\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0920f873-0f23-4dfb-9039-8404d4a83448\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt/\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"caab0c0b7118a7e434f4b749f1928e8e6d3893bf0055da03ec9c247054e3543e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1569, \"end_char_idx\": 4044, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2bb36a86-6fc3-4d31-a614-dad738a7899a", "doc_id": "2bb36a86-6fc3-4d31-a614-dad738a7899a", "ref_doc_id": "2bb36a86-6fc3-4d31-a614-dad738a7899a", "chroma:document": "Pre-trained on \nImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, \na crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks.*\n\nThis model was contributed by [anugunj](https://huggingface.co/anugunj). The original code can be found [here](https://github.com/microsoft/CvT).\n\n## Usage tips\n\n- CvT models are regular Vision Transformers, but trained with convolutions. They outperform the [original model (ViT)](vit) when fine-tuned on ImageNet-1K and CIFAR-100.\n- You can check out demo notebooks regarding inference as well as fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace [`ViTFeatureExtractor`] by [`AutoImageProcessor`] and [`ViTForImageClassification`] by [`CvtForImageClassification`]).\n- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of 14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CvT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`CvtForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## CvtConfig\n\n[[autodoc]] CvtConfig\n\n<frameworkcontent>\n<pt>\n\n## CvtModel\n\n[[autodoc]] CvtModel\n    - forward\n\n## CvtForImageClassification\n\n[[autodoc]] CvtForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFCvtModel\n\n[[autodoc]] TFCvtModel\n    - call\n\n## TFCvtForImageClassification\n\n[[autodoc]] TFCvtForImageClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 639bdf43-b55f-45ef-9db4-8019dc04a76a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinov2/", "title": "DINOv2", "tokens": 1171, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"639bdf43-b55f-45ef-9db4-8019dc04a76a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2/\", \"title\": \"DINOv2\", \"tokens\": 1171, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6846b1e3-08cc-4479-a769-0043cb744154\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2/\", \"title\": \"DINOv2\", \"tokens\": 1171, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e5058db4df5452e228642fd37f5034232d37efa5085202338eb7fa6419e60734\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b4175724-ddfd-4b9c-96e0-6ed880452f82\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ded34696c8b883b766e71796f6893c57bd9f4f451b03d9541883ac8fc42254c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2517, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6846b1e3-08cc-4479-a769-0043cb744154", "doc_id": "6846b1e3-08cc-4479-a769-0043cb744154", "ref_doc_id": "6846b1e3-08cc-4479-a769-0043cb744154", "chroma:document": "# DINOv2\n\n## Overview\n\nThe DINOv2 model was proposed in [DINOv2: Learning Robust Visual Features without Supervision](https://arxiv.org/abs/2304.07193) by\nMaxime Oquab, Timoth\u00e9e Darcet, Th\u00e9o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herv\u00e9 Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, Piotr Bojanowski.\nDINOv2 is an upgrade of [DINO](https://arxiv.org/abs/2104.14294), a self-supervised method applied on [Vision Transformers](vit). This method enables all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning.\n\nThe abstract from the paper is the following:\n\n*The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/facebookresearch/dinov2).\n\n## Usage tips\n\nThe model can be traced using `torch.jit.trace` which leverages JIT compilation to optimize the model making it faster to run. Note this still produces some mis-matched elements and the difference between the original model and the traced model is of the order of 1e-4."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b4175724-ddfd-4b9c-96e0-6ed880452f82
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinov2/", "title": "DINOv2", "tokens": 1171, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b4175724-ddfd-4b9c-96e0-6ed880452f82\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2/\", \"title\": \"DINOv2\", \"tokens\": 1171, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6846b1e3-08cc-4479-a769-0043cb744154\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2/\", \"title\": \"DINOv2\", \"tokens\": 1171, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e5058db4df5452e228642fd37f5034232d37efa5085202338eb7fa6419e60734\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"639bdf43-b55f-45ef-9db4-8019dc04a76a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2/\", \"title\": \"DINOv2\", \"tokens\": 1171, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"61f0a51df40cfcd62fe424c099b6d4b0bffd98d8018629fb72aedd6a6b6703e8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b61a4bf3-20d0-4d0b-86bf-92c8779b8aab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f18170280890c0e0996215e1a05283a32d306cfde57022d8a984949639ab0198\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 681, \"end_char_idx\": 4183, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6846b1e3-08cc-4479-a769-0043cb744154", "doc_id": "6846b1e3-08cc-4479-a769-0043cb744154", "ref_doc_id": "6846b1e3-08cc-4479-a769-0043cb744154", "chroma:document": "This method enables all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning.\n\nThe abstract from the paper is the following:\n\n*The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/facebookresearch/dinov2).\n\n## Usage tips\n\nThe model can be traced using `torch.jit.trace` which leverages JIT compilation to optimize the model making it faster to run. Note this still produces some mis-matched elements and the difference between the original model and the traced model is of the order of 1e-4.\n\n```python\nimport torch\nfrom transformers import AutoImageProcessor, AutoModel\nfrom PIL import Image\nimport requests\n\nurl = 'http://images.cocodataset.org/val2017/000000039769.jpg'\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprocessor = AutoImageProcessor.from_pretrained('facebook/dinov2-base')\nmodel = AutoModel.from_pretrained('facebook/dinov2-base')\n\ninputs = processor(images=image, return_tensors=\"pt\")\noutputs = model(**inputs)\nlast_hidden_states = outputs[0]\n\n# We have to force return_dict=False for tracing\nmodel.config.return_dict = False\n\nwith torch.no_grad():\n    traced_model = torch.jit.trace(model, [inputs.pixel_values])\n    traced_outputs = traced_model(inputs.pixel_values)\n\nprint((last_hidden_states - traced_outputs[0]).abs().max())\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DINOv2.\n\n- Demo notebooks for DINOv2 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DINOv2). \ud83c\udf0e\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`Dinov2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b61a4bf3-20d0-4d0b-86bf-92c8779b8aab
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinov2/", "title": "DINOv2", "tokens": 1171, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b61a4bf3-20d0-4d0b-86bf-92c8779b8aab\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2/\", \"title\": \"DINOv2\", \"tokens\": 1171, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6846b1e3-08cc-4479-a769-0043cb744154\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2/\", \"title\": \"DINOv2\", \"tokens\": 1171, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e5058db4df5452e228642fd37f5034232d37efa5085202338eb7fa6419e60734\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b4175724-ddfd-4b9c-96e0-6ed880452f82\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2/\", \"title\": \"DINOv2\", \"tokens\": 1171, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a37b7a7f49001b4f216e06a9fc4792d5b7899b9da18aca2e8655642c87b071cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2519, \"end_char_idx\": 4370, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6846b1e3-08cc-4479-a769-0043cb744154", "doc_id": "6846b1e3-08cc-4479-a769-0043cb744154", "ref_doc_id": "6846b1e3-08cc-4479-a769-0043cb744154", "chroma:document": "```python\nimport torch\nfrom transformers import AutoImageProcessor, AutoModel\nfrom PIL import Image\nimport requests\n\nurl = 'http://images.cocodataset.org/val2017/000000039769.jpg'\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprocessor = AutoImageProcessor.from_pretrained('facebook/dinov2-base')\nmodel = AutoModel.from_pretrained('facebook/dinov2-base')\n\ninputs = processor(images=image, return_tensors=\"pt\")\noutputs = model(**inputs)\nlast_hidden_states = outputs[0]\n\n# We have to force return_dict=False for tracing\nmodel.config.return_dict = False\n\nwith torch.no_grad():\n    traced_model = torch.jit.trace(model, [inputs.pixel_values])\n    traced_outputs = traced_model(inputs.pixel_values)\n\nprint((last_hidden_states - traced_outputs[0]).abs().max())\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DINOv2.\n\n- Demo notebooks for DINOv2 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DINOv2). \ud83c\udf0e\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`Dinov2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## Dinov2Config\n\n[[autodoc]] Dinov2Config\n\n## Dinov2Model\n\n[[autodoc]] Dinov2Model\n    - forward\n\n## Dinov2ForImageClassification\n\n[[autodoc]] Dinov2ForImageClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2911c304-ebda-46b8-b1cf-d45119ada96f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/univnet/", "title": "UnivNet", "tokens": 1204, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2911c304-ebda-46b8-b1cf-d45119ada96f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet/\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b9325da9-aa5b-4d25-8f87-3ca0535de834\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet/\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"23d03b33cfc2e5bff26c1b3a1cbf964c49664c142f5c36ded33c8d4e90dd644a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0600c425-609b-42a0-a295-880808f95aee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"643cc3abefefb7f891bdb15990ae11608f90361a279e08465d2c63081fbc079e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3045, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b9325da9-aa5b-4d25-8f87-3ca0535de834", "doc_id": "b9325da9-aa5b-4d25-8f87-3ca0535de834", "ref_doc_id": "b9325da9-aa5b-4d25-8f87-3ca0535de834", "chroma:document": "# UnivNet\n\n## Overview\n\nThe UnivNet model was proposed in [UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation](https://arxiv.org/abs/2106.07889) by Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kin, and Juntae Kim.\nThe UnivNet model is a generative adversarial network (GAN) trained to synthesize high fidelity speech waveforms. The UnivNet model shared in `transformers` is the *generator*, which maps a conditioning log-mel spectrogram and optional noise sequence to a speech waveform (e.g. a vocoder). Only the generator is required for inference. The *discriminator* used to train the `generator` is not implemented.\n\nThe abstract from the paper is the following:\n\n*Most neural vocoders employ band-limited mel-spectrograms to generate waveforms. If full-band spectral features are used as the input, the vocoder can be provided with as much acoustic information as possible. However, in some models employing full-band mel-spectrograms, an over-smoothing problem occurs as part of which non-sharp spectrograms are generated. To address this problem, we propose UnivNet, a neural vocoder that synthesizes high-fidelity waveforms in real time. Inspired by works in the field of voice activity detection, we added a multi-resolution spectrogram discriminator that employs multiple linear spectrogram magnitudes computed using various parameter sets. Using full-band mel-spectrograms as input, we expect to generate high-resolution signals by adding a discriminator that employs spectrograms of multiple resolutions as the input. In an evaluation on a dataset containing information on hundreds of speakers, UnivNet obtained the best objective and subjective results among competing models for both seen and unseen speakers. These results, including the best subjective score for text-to-speech, demonstrate the potential for fast adaptation to new speakers without a need for training from scratch.*\n\nTips:\n\n- The `noise_sequence` argument for [`UnivNetModel.forward`] should be standard Gaussian noise (such as from `torch.randn`) of shape `([batch_size], noise_length, model.config.model_in_channels)`, where `noise_length` should match the length dimension (dimension 1) of the `input_features` argument. If not supplied, it will be randomly generated; a `torch.Generator` can be supplied to the `generator` argument so that the forward pass can be reproduced. (Note that [`UnivNetFeatureExtractor`] will return generated noise by default, so it shouldn't be necessary to generate `noise_sequence` manually.)\n- Padding added by [`UnivNetFeatureExtractor`] can be removed from the [`UnivNetModel`] output through the [`UnivNetFeatureExtractor.batch_decode`] method, as shown in the usage example below.\n- Padding the end of each waveform with silence can reduce artifacts at the end of the generated audio sample. This can be done by supplying `pad_end = True` to [`UnivNetFeatureExtractor.__call__`]. See [this issue](https://github.com/seungwonpark/melgan/issues/8) for more details."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0600c425-609b-42a0-a295-880808f95aee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/univnet/", "title": "UnivNet", "tokens": 1204, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0600c425-609b-42a0-a295-880808f95aee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet/\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b9325da9-aa5b-4d25-8f87-3ca0535de834\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet/\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"23d03b33cfc2e5bff26c1b3a1cbf964c49664c142f5c36ded33c8d4e90dd644a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2911c304-ebda-46b8-b1cf-d45119ada96f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet/\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2a0dd9bfd52afaabe922115c65ba9aec8fac34b8825e62fb4dc0603a54d76a33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1203, \"end_char_idx\": 4646, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b9325da9-aa5b-4d25-8f87-3ca0535de834", "doc_id": "b9325da9-aa5b-4d25-8f87-3ca0535de834", "ref_doc_id": "b9325da9-aa5b-4d25-8f87-3ca0535de834", "chroma:document": "Inspired by works in the field of voice activity detection, we added a multi-resolution spectrogram discriminator that employs multiple linear spectrogram magnitudes computed using various parameter sets. Using full-band mel-spectrograms as input, we expect to generate high-resolution signals by adding a discriminator that employs spectrograms of multiple resolutions as the input. In an evaluation on a dataset containing information on hundreds of speakers, UnivNet obtained the best objective and subjective results among competing models for both seen and unseen speakers. These results, including the best subjective score for text-to-speech, demonstrate the potential for fast adaptation to new speakers without a need for training from scratch.*\n\nTips:\n\n- The `noise_sequence` argument for [`UnivNetModel.forward`] should be standard Gaussian noise (such as from `torch.randn`) of shape `([batch_size], noise_length, model.config.model_in_channels)`, where `noise_length` should match the length dimension (dimension 1) of the `input_features` argument. If not supplied, it will be randomly generated; a `torch.Generator` can be supplied to the `generator` argument so that the forward pass can be reproduced. (Note that [`UnivNetFeatureExtractor`] will return generated noise by default, so it shouldn't be necessary to generate `noise_sequence` manually.)\n- Padding added by [`UnivNetFeatureExtractor`] can be removed from the [`UnivNetModel`] output through the [`UnivNetFeatureExtractor.batch_decode`] method, as shown in the usage example below.\n- Padding the end of each waveform with silence can reduce artifacts at the end of the generated audio sample. This can be done by supplying `pad_end = True` to [`UnivNetFeatureExtractor.__call__`]. See [this issue](https://github.com/seungwonpark/melgan/issues/8) for more details.\n\nUsage Example:\n\n```python\nimport torch\nfrom scipy.io.wavfile import write\nfrom datasets import Audio, load_dataset\n\nfrom transformers import UnivNetFeatureExtractor, UnivNetModel\n\nmodel_id_or_path = \"dg845/univnet-dev\"\nmodel = UnivNetModel.from_pretrained(model_id_or_path)\nfeature_extractor = UnivNetFeatureExtractor.from_pretrained(model_id_or_path)\n\nds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n# Resample the audio to the model and feature extractor's sampling rate.\nds = ds.cast_column(\"audio\", Audio(sampling_rate=feature_extractor.sampling_rate))\n# Pad the end of the converted waveforms to reduce artifacts at the end of the output audio samples.\ninputs = feature_extractor(\n    ds[0][\"audio\"][\"array\"], sampling_rate=ds[0][\"audio\"][\"sampling_rate\"], pad_end=True, return_tensors=\"pt\"\n)\n\nwith torch.no_grad():\n    audio = model(**inputs)\n\n# Remove the extra padding at the end of the output.\naudio = feature_extractor.batch_decode(**audio)[0]\n# Convert to wav file\nwrite(\"sample_audio.wav\", feature_extractor.sampling_rate, audio)\n```\n\nThis model was contributed by [dg845](https://huggingface.co/dg845).\nTo the best of my knowledge, there is no official code release, but an unofficial implementation can be found at [maum-ai/univnet](https://github.com/maum-ai/univnet) with pretrained checkpoints [here](https://github.com/maum-ai/univnet#pre-trained-model).\n\n\n## UnivNetConfig\n\n[[autodoc]] UnivNetConfig\n\n## UnivNetFeatureExtractor\n\n[[autodoc]] UnivNetFeatureExtractor\n    - __call__\n\n## UnivNetModel\n\n[[autodoc]] UnivNetModel\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 365bb4ee-c075-498b-9221-9905eaef62e4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jukebox/", "title": "Jukebox", "tokens": 1168, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"365bb4ee-c075-498b-9221-9905eaef62e4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox/\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c54f65d-4f36-486b-9932-d120cb492008\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox/\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8d1297c41cd166f89e8b120fecffe425b18be8a932e78d799925a1474ead6d8c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8f07f4f4-7628-47ff-9e79-5997b96bcdf9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"86d88cefc8b377efbe78c222330b95399004ef1cddafd7550c2f8414715ff1f7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3150, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c54f65d-4f36-486b-9932-d120cb492008", "doc_id": "4c54f65d-4f36-486b-9932-d120cb492008", "ref_doc_id": "4c54f65d-4f36-486b-9932-d120cb492008", "chroma:document": "# Jukebox\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe Jukebox model was proposed in [Jukebox: A generative model for music](https://arxiv.org/pdf/2005.00341.pdf)\nby Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford,\nIlya Sutskever. It introduces a generative music model which can produce minute long samples that can be conditioned on\nan artist, genres and lyrics.\n\nThe abstract from the paper is the following:\n\n*We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multiscale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers. We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes. We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable. We are releasing thousands of non cherry-picked samples, along with model weights and code.*\n\nAs shown on the following figure, Jukebox is made of 3 `priors` which are decoder only models. They follow the architecture described in [Generating Long Sequences with Sparse Transformers](https://arxiv.org/abs/1904.10509), modified to support longer context length.\nFirst, a autoencoder is used to encode the text lyrics. Next, the first (also called `top_prior`) prior attends to the last hidden states extracted from the lyrics encoder. The priors are linked to the previous priors respectively via an `AudioConditioner` module. The`AudioConditioner` upsamples the outputs of the previous prior to raw tokens at a certain audio frame per second resolution.\nThe metadata such as *artist, genre and timing* are passed to each prior, in the form of a start token and positional embedding for the timing data.  The hidden states are mapped to the closest codebook vector from the VQVAE in order to convert them to raw audio.\n\n![JukeboxModel](https://gist.githubusercontent.com/ArthurZucker/92c1acaae62ebf1b6a951710bdd8b6af/raw/c9c517bf4eff61393f6c7dec9366ef02bdd059a3/jukebox.svg)\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/openai/jukebox).\n\n## Usage tips\n\n- This model only supports inference. This is for a few reasons, mostly because it requires a crazy amount of memory to train. Feel free to open a PR and add what's missing to have a full integration with the hugging face trainer!\n- This model is very slow, and takes 8h to generate a minute long audio using the 5b top prior on a V100 GPU. In order automaticallay handle the device on which the model should execute, use `accelerate`.\n- Contrary to the paper, the order of the priors goes from `0` to `1` as it felt more intuitive : we sample starting from `0`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8f07f4f4-7628-47ff-9e79-5997b96bcdf9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jukebox/", "title": "Jukebox", "tokens": 1168, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8f07f4f4-7628-47ff-9e79-5997b96bcdf9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox/\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c54f65d-4f36-486b-9932-d120cb492008\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox/\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8d1297c41cd166f89e8b120fecffe425b18be8a932e78d799925a1474ead6d8c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"365bb4ee-c075-498b-9221-9905eaef62e4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox/\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"23c30e98f1d1c2282754a6225cc7a6b2576d32b6136b11452616e22db7e8d11c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1605, \"end_char_idx\": 3972, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c54f65d-4f36-486b-9932-d120cb492008", "doc_id": "4c54f65d-4f36-486b-9932-d120cb492008", "ref_doc_id": "4c54f65d-4f36-486b-9932-d120cb492008", "chroma:document": "First, a autoencoder is used to encode the text lyrics. Next, the first (also called `top_prior`) prior attends to the last hidden states extracted from the lyrics encoder. The priors are linked to the previous priors respectively via an `AudioConditioner` module. The`AudioConditioner` upsamples the outputs of the previous prior to raw tokens at a certain audio frame per second resolution.\nThe metadata such as *artist, genre and timing* are passed to each prior, in the form of a start token and positional embedding for the timing data.  The hidden states are mapped to the closest codebook vector from the VQVAE in order to convert them to raw audio.\n\n![JukeboxModel](https://gist.githubusercontent.com/ArthurZucker/92c1acaae62ebf1b6a951710bdd8b6af/raw/c9c517bf4eff61393f6c7dec9366ef02bdd059a3/jukebox.svg)\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/openai/jukebox).\n\n## Usage tips\n\n- This model only supports inference. This is for a few reasons, mostly because it requires a crazy amount of memory to train. Feel free to open a PR and add what's missing to have a full integration with the hugging face trainer!\n- This model is very slow, and takes 8h to generate a minute long audio using the 5b top prior on a V100 GPU. In order automaticallay handle the device on which the model should execute, use `accelerate`.\n- Contrary to the paper, the order of the priors goes from `0` to `1` as it felt more intuitive : we sample starting from `0`.\n- Primed sampling (conditioning the sampling on raw audio) requires more memory than ancestral sampling and should be used with `fp16` set to `True`.\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/openai/jukebox).\n\n## JukeboxConfig\n\n[[autodoc]] JukeboxConfig\n\n## JukeboxPriorConfig\n\n[[autodoc]] JukeboxPriorConfig\n\n## JukeboxVQVAEConfig\n\n[[autodoc]] JukeboxVQVAEConfig\n\n## JukeboxTokenizer\n\n[[autodoc]] JukeboxTokenizer\n    - save_vocabulary\n\n## JukeboxModel\n\n[[autodoc]] JukeboxModel\n    - ancestral_sample\n    - primed_sample\n    - continue_sample\n    - upsample\n    - _sample\n\n## JukeboxPrior\n\n[[autodoc]] JukeboxPrior\n    - sample\n    - forward\n\n## JukeboxVQVAE\n\n[[autodoc]] JukeboxVQVAE\n    - forward\n    - encode\n    - decode"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f4cd9a6b-e895-4116-a248-4a4f29051956
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen/", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f4cd9a6b-e895-4116-a248-4a4f29051956\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ae8ac7-6fa0-4f17-8d68-130b8f4eb038\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"554328c56b3c24f64e8bf14dcadddbe9767458d87bc9c8d8b7dd9e424485400d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c66a2778-fa28-4c8a-99d2-182e06bfe26b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f03d8a21442981fccacce640760f6c4d570a984c750f06c5c3d1c3bf89c1e9b8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3437, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "ref_doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "chroma:document": "# MusicGen\n\n## Overview\n\nThe MusicGen model was proposed in the paper [Simple and Controllable Music Generation](https://arxiv.org/abs/2306.05284)\nby Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi and Alexandre D\u00e9fossez.\n\nMusicGen is a single stage auto-regressive Transformer model capable of generating high-quality music samples conditioned\non text descriptions or audio prompts. The text descriptions are passed through a frozen text encoder model to obtain a\nsequence of hidden-state representations. MusicGen is then trained to predict discrete audio tokens, or *audio codes*,\nconditioned on these hidden-states. These audio tokens are then decoded using an audio compression model, such as EnCodec,\nto recover the audio waveform.\n\nThrough an efficient token interleaving pattern, MusicGen does not require a self-supervised semantic representation of\nthe text/audio prompts, thus eliminating the need to cascade multiple models to predict a set of codebooks (e.g.\nhierarchically or upsampling). Instead, it is able to generate all the codebooks in a single forward pass.\n\nThe abstract from the paper is the following:\n\n*We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates\nover several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised\nof a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for\ncascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen\ncan generate high-quality samples, while being conditioned on textual description or melodic features, allowing better\ncontrols over the generated output. We conduct extensive empirical evaluation, considering both automatic and human\nstudies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark.\nThrough ablation studies, we shed light over the importance of each of the components comprising MusicGen.*\n\nThis model was contributed by [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found\n[here](https://github.com/facebookresearch/audiocraft). The pre-trained checkpoints can be found on the\n[Hugging Face Hub](https://huggingface.co/models?sort=downloads&search=facebook%2Fmusicgen-).\n\n## Usage tips\n\n- After downloading the original checkpoints from [here](https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md#importing--exporting-models) , you can convert them using the **conversion script** available at\n`src/transformers/models/musicgen/convert_musicgen_transformers.py` with the following command:\n\n```bash\npython src/transformers/models/musicgen/convert_musicgen_transformers.py \\\n    --checkpoint small --pytorch_dump_folder /output/path --safe_serialization \n```\n\n## Generation\n\nMusicGen is compatible with two generation modes: greedy and sampling. In practice, sampling leads to significantly\nbetter results than greedy, thus we encourage sampling mode to be used where possible. Sampling is enabled by default,\nand can be explicitly specified by setting `do_sample=True` in the call to [`MusicgenForConditionalGeneration.generate`],\nor by overriding the model's generation config (see below).\n\nGeneration is limited by the sinusoidal positional embeddings to 30 second inputs."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c66a2778-fa28-4c8a-99d2-182e06bfe26b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen/", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c66a2778-fa28-4c8a-99d2-182e06bfe26b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ae8ac7-6fa0-4f17-8d68-130b8f4eb038\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"554328c56b3c24f64e8bf14dcadddbe9767458d87bc9c8d8b7dd9e424485400d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f4cd9a6b-e895-4116-a248-4a4f29051956\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a7307901ddfb529cdd617745cc2501279ea504019a37e52a565413df238c9229\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d30dd718-50db-4992-a651-8d30da76c49e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b194ce0d6f527ef14b48684edac389226c4638735fbdcc4b72c949f5a0be0b05\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1583, \"end_char_idx\": 5050, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "ref_doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "chroma:document": "Following this approach, we demonstrate how MusicGen\ncan generate high-quality samples, while being conditioned on textual description or melodic features, allowing better\ncontrols over the generated output. We conduct extensive empirical evaluation, considering both automatic and human\nstudies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark.\nThrough ablation studies, we shed light over the importance of each of the components comprising MusicGen.*\n\nThis model was contributed by [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found\n[here](https://github.com/facebookresearch/audiocraft). The pre-trained checkpoints can be found on the\n[Hugging Face Hub](https://huggingface.co/models?sort=downloads&search=facebook%2Fmusicgen-).\n\n## Usage tips\n\n- After downloading the original checkpoints from [here](https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md#importing--exporting-models) , you can convert them using the **conversion script** available at\n`src/transformers/models/musicgen/convert_musicgen_transformers.py` with the following command:\n\n```bash\npython src/transformers/models/musicgen/convert_musicgen_transformers.py \\\n    --checkpoint small --pytorch_dump_folder /output/path --safe_serialization \n```\n\n## Generation\n\nMusicGen is compatible with two generation modes: greedy and sampling. In practice, sampling leads to significantly\nbetter results than greedy, thus we encourage sampling mode to be used where possible. Sampling is enabled by default,\nand can be explicitly specified by setting `do_sample=True` in the call to [`MusicgenForConditionalGeneration.generate`],\nor by overriding the model's generation config (see below).\n\nGeneration is limited by the sinusoidal positional embeddings to 30 second inputs. Meaning, MusicGen cannot generate more\nthan 30 seconds of audio (1503 tokens), and input audio passed by Audio-Prompted Generation contributes to this limit so,\ngiven an input of 20 seconds of audio, MusicGen cannot generate more than 10 seconds of additional audio.\n\nTransformers supports both mono (1-channel) and stereo (2-channel) variants of MusicGen. The mono channel versions \ngenerate a single set of codebooks. The stereo versions generate 2 sets of codebooks, 1 for each channel (left/right), \nand each set of codebooks is decoded independently through the audio compression model. The audio streams for each \nchannel are combined to give the final stereo output.\n\n### Unconditional Generation\n\nThe inputs for unconditional (or 'null') generation can be obtained through the method\n[`MusicgenForConditionalGeneration.get_unconditional_inputs`]:\n\n```python\n>>> from transformers import MusicgenForConditionalGeneration\n\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n>>> unconditional_inputs = model.get_unconditional_inputs(num_samples=1)\n\n>>> audio_values = model.generate(**unconditional_inputs, do_sample=True, max_new_tokens=256)\n```\n\nThe audio outputs are a three-dimensional Torch tensor of shape `(batch_size, num_channels, sequence_length)`. To listen\nto the generated audio samples, you can either play them in an ipynb notebook:\n\n```python\nfrom IPython.display import Audio\n\nsampling_rate = model.config.audio_encoder.sampling_rate\nAudio(audio_values[0].numpy(), rate=sampling_rate)\n```\n\nOr save them as a `.wav` file using a third-party library, e.g."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d30dd718-50db-4992-a651-8d30da76c49e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen/", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d30dd718-50db-4992-a651-8d30da76c49e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ae8ac7-6fa0-4f17-8d68-130b8f4eb038\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"554328c56b3c24f64e8bf14dcadddbe9767458d87bc9c8d8b7dd9e424485400d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c66a2778-fa28-4c8a-99d2-182e06bfe26b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"18f1edfde462a975102f108df538273aa1960ad31fb944d027afbecb2194fb9d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0321b40-6798-452d-8f60-1f29171dc3f8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0304c92bc724c8d89aa7d5f8a9491ecd66c6ad340cd4a22e8571d3a60122a890\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3355, \"end_char_idx\": 6624, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "ref_doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "chroma:document": "Generation is limited by the sinusoidal positional embeddings to 30 second inputs. Meaning, MusicGen cannot generate more\nthan 30 seconds of audio (1503 tokens), and input audio passed by Audio-Prompted Generation contributes to this limit so,\ngiven an input of 20 seconds of audio, MusicGen cannot generate more than 10 seconds of additional audio.\n\nTransformers supports both mono (1-channel) and stereo (2-channel) variants of MusicGen. The mono channel versions \ngenerate a single set of codebooks. The stereo versions generate 2 sets of codebooks, 1 for each channel (left/right), \nand each set of codebooks is decoded independently through the audio compression model. The audio streams for each \nchannel are combined to give the final stereo output.\n\n### Unconditional Generation\n\nThe inputs for unconditional (or 'null') generation can be obtained through the method\n[`MusicgenForConditionalGeneration.get_unconditional_inputs`]:\n\n```python\n>>> from transformers import MusicgenForConditionalGeneration\n\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n>>> unconditional_inputs = model.get_unconditional_inputs(num_samples=1)\n\n>>> audio_values = model.generate(**unconditional_inputs, do_sample=True, max_new_tokens=256)\n```\n\nThe audio outputs are a three-dimensional Torch tensor of shape `(batch_size, num_channels, sequence_length)`. To listen\nto the generated audio samples, you can either play them in an ipynb notebook:\n\n```python\nfrom IPython.display import Audio\n\nsampling_rate = model.config.audio_encoder.sampling_rate\nAudio(audio_values[0].numpy(), rate=sampling_rate)\n```\n\nOr save them as a `.wav` file using a third-party library, e.g. `scipy`:\n\n```python\n>>> import scipy\n\n>>> sampling_rate = model.config.audio_encoder.sampling_rate\n>>> scipy.io.wavfile.write(\"musicgen_out.wav\", rate=sampling_rate, data=audio_values[0, 0].numpy())\n```\n\n### Text-Conditional Generation\n\nThe model can generate an audio sample conditioned on a text prompt through use of the [`MusicgenProcessor`] to pre-process\nthe inputs:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-small\")\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> inputs = processor(\n...     text=[\"80s pop track with bassy drums and synth\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe `guidance_scale` is used in classifier free guidance (CFG), setting the weighting between the conditional logits\n(which are predicted from the text prompts) and the unconditional logits (which are predicted from an unconditional or\n'null' prompt). Higher guidance scale encourages the model to generate samples that are more closely linked to the input\nprompt, usually at the expense of poorer audio quality. CFG is enabled by setting `guidance_scale > 1`. For best results,\nuse `guidance_scale=3` (default).\n\n### Audio-Prompted Generation\n\nThe same [`MusicgenProcessor`] can be used to pre-process an audio prompt that is used for audio continuation."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e0321b40-6798-452d-8f60-1f29171dc3f8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen/", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e0321b40-6798-452d-8f60-1f29171dc3f8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ae8ac7-6fa0-4f17-8d68-130b8f4eb038\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"554328c56b3c24f64e8bf14dcadddbe9767458d87bc9c8d8b7dd9e424485400d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d30dd718-50db-4992-a651-8d30da76c49e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"aecbff09f22d5d7edc2a52fbd9c45d42367ad7381f65a38dc34d8d1c5489a2b1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2a160a0e-8dcc-4d10-b55e-033a75edb75c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e21ef78a833b3c0679c884f40d690f9f6c1681ba2a6855a70b21a569dbef81e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5051, \"end_char_idx\": 9875, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "ref_doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "chroma:document": "`scipy`:\n\n```python\n>>> import scipy\n\n>>> sampling_rate = model.config.audio_encoder.sampling_rate\n>>> scipy.io.wavfile.write(\"musicgen_out.wav\", rate=sampling_rate, data=audio_values[0, 0].numpy())\n```\n\n### Text-Conditional Generation\n\nThe model can generate an audio sample conditioned on a text prompt through use of the [`MusicgenProcessor`] to pre-process\nthe inputs:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-small\")\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> inputs = processor(\n...     text=[\"80s pop track with bassy drums and synth\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe `guidance_scale` is used in classifier free guidance (CFG), setting the weighting between the conditional logits\n(which are predicted from the text prompts) and the unconditional logits (which are predicted from an unconditional or\n'null' prompt). Higher guidance scale encourages the model to generate samples that are more closely linked to the input\nprompt, usually at the expense of poorer audio quality. CFG is enabled by setting `guidance_scale > 1`. For best results,\nuse `guidance_scale=3` (default).\n\n### Audio-Prompted Generation\n\nThe same [`MusicgenProcessor`] can be used to pre-process an audio prompt that is used for audio continuation. In the\nfollowing example, we load an audio file using the \ud83e\udd17 Datasets library, which can be pip installed through the command\nbelow:\n\n```bash\npip install --upgrade pip\npip install datasets[audio]\n```\n\n```python\n>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-small\")\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> dataset = load_dataset(\"sanchit-gandhi/gtzan\", split=\"train\", streaming=True)\n>>> sample = next(iter(dataset))[\"audio\"]\n\n>>> # take the first half of the audio sample\n>>> sample[\"array\"] = sample[\"array\"][: len(sample[\"array\"]) // 2]\n\n>>> inputs = processor(\n...     audio=sample[\"array\"],\n...     sampling_rate=sample[\"sampling_rate\"],\n...     text=[\"80s blues track with groovy saxophone\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nFor batched audio-prompted generation, the generated `audio_values` can be post-processed to remove padding by using the\n[`MusicgenProcessor`] class:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-small\")\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> dataset = load_dataset(\"sanchit-gandhi/gtzan\", split=\"train\", streaming=True)\n>>> sample = next(iter(dataset))[\"audio\"]\n\n>>> # take the first quarter of the audio sample\n>>> sample_1 = sample[\"array\"][: len(sample[\"array\"]) // 4]\n\n>>> # take the first half of the audio sample\n>>> sample_2 = sample[\"array\"][: len(sample[\"array\"]) // 2]\n\n>>> inputs = processor(\n...     audio=[sample_1, sample_2],\n...     sampling_rate=sample[\"sampling_rate\"],\n...     text=[\"80s blues track with groovy saxophone\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n\n>>> # post-process to remove padding from the batched audio\n>>> audio_values = processor.batch_decode(audio_values, padding_mask=inputs.padding_mask)\n```\n\n### Generation Configuration\n\nThe default parameters that control the generation process, such as sampling, guidance scale and number of generated \ntokens, can be found in the model's generation config, and updated as desired:\n\n```python\n>>> from transformers import MusicgenForConditionalGeneration\n\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> # inspect the default generation config\n>>> model.generation_config\n\n>>> # increase the guidance scale to 4.0\n>>> model.generation_config.guidance_scale = 4.0\n\n>>> # decrease the max length to 256 tokens\n>>> model.generation_config.max_length = 256\n```\n\nNote that any arguments passed to the generate method will **supersede** those in the generation config, so setting \n`do_sample=False` in the call to generate will supersede the setting of `model.generation_config.do_sample` in the \ngeneration config."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2a160a0e-8dcc-4d10-b55e-033a75edb75c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen/", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2a160a0e-8dcc-4d10-b55e-033a75edb75c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ae8ac7-6fa0-4f17-8d68-130b8f4eb038\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"554328c56b3c24f64e8bf14dcadddbe9767458d87bc9c8d8b7dd9e424485400d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0321b40-6798-452d-8f60-1f29171dc3f8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen/\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"96314794dc76f774eff0e76c40bc95e26cc3d5b193f369a484abe0ec8cad06c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9877, \"end_char_idx\": 12541, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "ref_doc_id": "59ae8ac7-6fa0-4f17-8d68-130b8f4eb038", "chroma:document": "## Model Structure\n\nThe MusicGen model can be de-composed into three distinct stages:\n1. Text encoder: maps the text inputs to a sequence of hidden-state representations. The pre-trained MusicGen models use a frozen text encoder from either T5 or Flan-T5\n2. MusicGen decoder: a language model (LM) that auto-regressively generates audio tokens (or codes) conditional on the encoder hidden-state representations\n3. Audio encoder/decoder: used to encode an audio prompt to use as prompt tokens, and recover the audio waveform from the audio tokens predicted by the decoder\n\nThus, the MusicGen model can either be used as a standalone decoder model, corresponding to the class [`MusicgenForCausalLM`],\nor as a composite model that includes the text encoder and audio encoder/decoder, corresponding to the class\n[`MusicgenForConditionalGeneration`]. If only the decoder needs to be loaded from the pre-trained checkpoint, it can be loaded by first \nspecifying the correct config, or be accessed through the `.decoder` attribute of the composite model:\n\n```python\n>>> from transformers import AutoConfig, MusicgenForCausalLM, MusicgenForConditionalGeneration\n\n>>> # Option 1: get decoder config and pass to `.from_pretrained`\n>>> decoder_config = AutoConfig.from_pretrained(\"facebook/musicgen-small\").decoder\n>>> decoder = MusicgenForCausalLM.from_pretrained(\"facebook/musicgen-small\", **decoder_config)\n\n>>> # Option 2: load the entire composite model, but only return the decoder\n>>> decoder = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\").decoder\n```\n\nSince the text encoder and audio encoder/decoder models are frozen during training, the MusicGen decoder [`MusicgenForCausalLM`]\ncan be trained standalone on a dataset of encoder hidden-states and audio codes. For inference, the trained decoder can\nbe combined with the frozen text encoder and audio encoder/decoders to recover the composite [`MusicgenForConditionalGeneration`]\nmodel.\n\nTips:\n* MusicGen is trained on the 32kHz checkpoint of Encodec. You should ensure you use a compatible version of the Encodec model.\n* Sampling mode tends to deliver better results than greedy - you can toggle sampling with the variable `do_sample` in the call to [`MusicgenForConditionalGeneration.generate`]\n\n## MusicgenDecoderConfig\n\n[[autodoc]] MusicgenDecoderConfig\n\n## MusicgenConfig\n\n[[autodoc]] MusicgenConfig\n\n## MusicgenProcessor\n\n[[autodoc]] MusicgenProcessor\n\n## MusicgenModel\n\n[[autodoc]] MusicgenModel\n    - forward\n\n## MusicgenForCausalLM\n\n[[autodoc]] MusicgenForCausalLM\n    - forward\n\n## MusicgenForConditionalGeneration\n\n[[autodoc]] MusicgenForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b3d63fd6-2c15-4229-8c8f-537d01dc907c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swin/", "title": "Swin Transformer", "tokens": 1163, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b3d63fd6-2c15-4229-8c8f-537d01dc907c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin/\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3c3999cb-936a-42c6-946c-6f52dfd40099\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin/\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"eff77bb88f97e9b4f5d162576abaa3191c6f1ae25f1c31856e1c755a8cb598cc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"408e5d0b-0498-4375-87bd-b46abd2f7c0a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6cc4e1c3227f42ca5052ed670afcacce19990da0626534ab887f1f49701ebf5e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3191, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3c3999cb-936a-42c6-946c-6f52dfd40099", "doc_id": "3c3999cb-936a-42c6-946c-6f52dfd40099", "ref_doc_id": "3c3999cb-936a-42c6-946c-6f52dfd40099", "chroma:document": "# Swin Transformer\n\n## Overview\n\nThe Swin Transformer was proposed in [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://arxiv.org/abs/2103.14030)\nby Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo.\n\nThe abstract from the paper is the following:\n\n*This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone\nfor computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains,\nsuch as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text.\nTo address these differences, we propose a hierarchical Transformer whose representation is computed with \\bold{S}hifted\n\\bold{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping\nlocal windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at\nvarious scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it\ncompatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense\nprediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation\n(53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and\n+2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones.\nThe hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/swin_transformer_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Swin Transformer architecture. Taken from the <a href=\"https://arxiv.org/abs/2102.03334\">original paper</a>.</small>\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The Tensorflow version of this model was contributed by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/microsoft/Swin-Transformer).\n\n## Usage tips\n\n- Swin pads the inputs supporting any input height and width (if divisible by `32`).\n- Swin can be used as a *backbone*. When `output_hidden_states = True`, it will output both `hidden_states` and `reshaped_hidden_states`. The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than `(batch_size, sequence_length, num_channels)`.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Swin Transformer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`SwinForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 408e5d0b-0498-4375-87bd-b46abd2f7c0a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swin/", "title": "Swin Transformer", "tokens": 1163, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"408e5d0b-0498-4375-87bd-b46abd2f7c0a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin/\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3c3999cb-936a-42c6-946c-6f52dfd40099\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin/\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"eff77bb88f97e9b4f5d162576abaa3191c6f1ae25f1c31856e1c755a8cb598cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b3d63fd6-2c15-4229-8c8f-537d01dc907c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin/\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7278ebb6fa65225cbc308f2486b8be54510ffd348126a6b668c6e90b579eb97b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1683, \"end_char_idx\": 4248, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3c3999cb-936a-42c6-946c-6f52dfd40099", "doc_id": "3c3999cb-936a-42c6-946c-6f52dfd40099", "ref_doc_id": "3c3999cb-936a-42c6-946c-6f52dfd40099", "chroma:document": "The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/swin_transformer_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Swin Transformer architecture. Taken from the <a href=\"https://arxiv.org/abs/2102.03334\">original paper</a>.</small>\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The Tensorflow version of this model was contributed by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/microsoft/Swin-Transformer).\n\n## Usage tips\n\n- Swin pads the inputs supporting any input height and width (if divisible by `32`).\n- Swin can be used as a *backbone*. When `output_hidden_states = True`, it will output both `hidden_states` and `reshaped_hidden_states`. The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than `(batch_size, sequence_length, num_channels)`.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Swin Transformer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`SwinForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nBesides that:\n\n- [`SwinForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## SwinConfig\n\n[[autodoc]] SwinConfig\n\n<frameworkcontent>\n<pt>\n\n## SwinModel\n\n[[autodoc]] SwinModel\n    - forward\n\n## SwinForMaskedImageModeling\n\n[[autodoc]] SwinForMaskedImageModeling\n    - forward\n\n## SwinForImageClassification\n\n[[autodoc]] transformers.SwinForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFSwinModel\n\n[[autodoc]] TFSwinModel\n    - call\n\n## TFSwinForMaskedImageModeling\n\n[[autodoc]] TFSwinForMaskedImageModeling\n    - call\n\n## TFSwinForImageClassification\n\n[[autodoc]] transformers.TFSwinForImageClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e495ab32-fdad-47fa-9040-b6ff90228067
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/perceiver/", "title": "Perceiver", "tokens": 2357, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e495ab32-fdad-47fa-9040-b6ff90228067\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5804f68f-131a-44e4-b27a-214bc37d2455\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ee7a2699c6d096f65e9c99af40d88a286baf4ff7fcc28d6039fdbc13442c950c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"94253a2b-e69b-42cd-a189-6d5cb08f384f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"21dcf02be4582a271e345a9146bea269ad9d5399be2f717621481db84bc40889\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3415, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "doc_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "ref_doc_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "chroma:document": "# Perceiver\n\n## Overview\n\nThe Perceiver IO model was proposed in [Perceiver IO: A General Architecture for Structured Inputs &\nOutputs](https://arxiv.org/abs/2107.14795) by Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch,\nCatalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier H\u00e9naff, Matthew M.\nBotvinick, Andrew Zisserman, Oriol Vinyals, Jo\u00e3o Carreira.\n\nPerceiver IO is a generalization of [Perceiver](https://arxiv.org/abs/2103.03206) to handle arbitrary outputs in\naddition to arbitrary inputs. The original Perceiver only produced a single classification label. In addition to\nclassification labels, Perceiver IO can produce (for example) language, optical flow, and multimodal videos with audio.\nThis is done using the same building blocks as the original Perceiver. The computational complexity of Perceiver IO is\nlinear in the input and output size and the bulk of the processing occurs in the latent space, allowing us to process\ninputs and outputs that are much larger than can be handled by standard Transformers. This means, for example,\nPerceiver IO can do BERT-style masked language modeling directly using bytes instead of tokenized inputs.\n\nThe abstract from the paper is the following:\n\n*The recently-proposed Perceiver model obtains good results on several domains (images, audio, multimodal, point\nclouds) while scaling linearly in compute and memory with the input size. While the Perceiver supports many kinds of\ninputs, it can only produce very simple outputs such as class scores. Perceiver IO overcomes this limitation without\nsacrificing the original's appealing properties by learning to flexibly query the model's latent space to produce\noutputs of arbitrary size and semantics. Perceiver IO still decouples model depth from data size and still scales\nlinearly with data size, but now with respect to both input and output sizes. The full Perceiver IO model achieves\nstrong results on tasks with highly structured output spaces, such as natural language and visual understanding,\nStarCraft II, and multi-task and multi-modal domains. As highlights, Perceiver IO matches a Transformer-based BERT\nbaseline on the GLUE language benchmark without the need for input tokenization and achieves state-of-the-art\nperformance on Sintel optical flow estimation.*\n\nHere's a TLDR explaining how Perceiver works:\n\nThe main problem with the self-attention mechanism of the Transformer is that the time and memory requirements scale\nquadratically with the sequence length. Hence, models like BERT and RoBERTa are limited to a max sequence length of 512\ntokens. Perceiver aims to solve this issue by, instead of performing self-attention on the inputs, perform it on a set\nof latent variables, and only use the inputs for cross-attention. In this way, the time and memory requirements don't\ndepend on the length of the inputs anymore, as one uses a fixed amount of latent variables, like 256 or 512. These are\nrandomly initialized, after which they are trained end-to-end using backpropagation.\n\nInternally, [`PerceiverModel`] will create the latents, which is a tensor of shape `(batch_size, num_latents,\nd_latents)`. One must provide `inputs` (which could be text, images, audio, you name it!) to the model, which it will\nuse to perform cross-attention with the latents. The output of the Perceiver encoder is a tensor of the same shape."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 94253a2b-e69b-42cd-a189-6d5cb08f384f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/perceiver/", "title": "Perceiver", "tokens": 2357, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"94253a2b-e69b-42cd-a189-6d5cb08f384f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5804f68f-131a-44e4-b27a-214bc37d2455\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ee7a2699c6d096f65e9c99af40d88a286baf4ff7fcc28d6039fdbc13442c950c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e495ab32-fdad-47fa-9040-b6ff90228067\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b69527ee4addd5aa2371539e500b218f26097f16499dfd320b36a667a7f12e7c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"31215f69-6570-4011-83c2-2c56a5abea24\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b35577626ee74d83e858dab400e8e7a60824d7e59cd609404732b3083dae7514\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1771, \"end_char_idx\": 4942, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "doc_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "ref_doc_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "chroma:document": "Perceiver IO still decouples model depth from data size and still scales\nlinearly with data size, but now with respect to both input and output sizes. The full Perceiver IO model achieves\nstrong results on tasks with highly structured output spaces, such as natural language and visual understanding,\nStarCraft II, and multi-task and multi-modal domains. As highlights, Perceiver IO matches a Transformer-based BERT\nbaseline on the GLUE language benchmark without the need for input tokenization and achieves state-of-the-art\nperformance on Sintel optical flow estimation.*\n\nHere's a TLDR explaining how Perceiver works:\n\nThe main problem with the self-attention mechanism of the Transformer is that the time and memory requirements scale\nquadratically with the sequence length. Hence, models like BERT and RoBERTa are limited to a max sequence length of 512\ntokens. Perceiver aims to solve this issue by, instead of performing self-attention on the inputs, perform it on a set\nof latent variables, and only use the inputs for cross-attention. In this way, the time and memory requirements don't\ndepend on the length of the inputs anymore, as one uses a fixed amount of latent variables, like 256 or 512. These are\nrandomly initialized, after which they are trained end-to-end using backpropagation.\n\nInternally, [`PerceiverModel`] will create the latents, which is a tensor of shape `(batch_size, num_latents,\nd_latents)`. One must provide `inputs` (which could be text, images, audio, you name it!) to the model, which it will\nuse to perform cross-attention with the latents. The output of the Perceiver encoder is a tensor of the same shape. One\ncan then, similar to BERT, convert the last hidden states of the latents to classification logits by averaging along\nthe sequence dimension, and placing a linear layer on top of that to project the `d_latents` to `num_labels`.\n\nThis was the idea of the original Perceiver paper. However, it could only output classification logits. In a follow-up\nwork, PerceiverIO, they generalized it to let the model also produce outputs of arbitrary size. How, you might ask? The\nidea is actually relatively simple: one defines outputs of an arbitrary size, and then applies cross-attention with the\nlast hidden states of the latents, using the outputs as queries, and the latents as keys and values.\n\nSo let's say one wants to perform masked language modeling (BERT-style) with the Perceiver. As the Perceiver's input\nlength will not have an impact on the computation time of the self-attention layers, one can provide raw bytes,\nproviding `inputs` of length 2048 to the model. If one now masks out certain of these 2048 tokens, one can define the\n`outputs` as being of shape: `(batch_size, 2048, 768)`. Next, one performs cross-attention with the final hidden states\nof the latents to update the `outputs` tensor. After cross-attention, one still has a tensor of shape `(batch_size,\n2048, 768)`. One can then place a regular language modeling head on top, to project the last dimension to the\nvocabulary size of the model, i.e. creating logits of shape `(batch_size, 2048, 262)` (as Perceiver uses a vocabulary\nsize of 262 byte IDs)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 31215f69-6570-4011-83c2-2c56a5abea24
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/perceiver/", "title": "Perceiver", "tokens": 2357, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"31215f69-6570-4011-83c2-2c56a5abea24\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5804f68f-131a-44e4-b27a-214bc37d2455\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ee7a2699c6d096f65e9c99af40d88a286baf4ff7fcc28d6039fdbc13442c950c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"94253a2b-e69b-42cd-a189-6d5cb08f384f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e246cc360415c7d125c9ca2e53928e35cbc83d72064b95f6899ba46e236f8e4d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"81130b3e-0143-4aa9-ab6a-e827e08baad2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7ae2dbbc2c922f8330c89be1b3c6a7da4bcb4699b6b8d01282c1963b79175f6b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3349, \"end_char_idx\": 6177, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "doc_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "ref_doc_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "chroma:document": "The output of the Perceiver encoder is a tensor of the same shape. One\ncan then, similar to BERT, convert the last hidden states of the latents to classification logits by averaging along\nthe sequence dimension, and placing a linear layer on top of that to project the `d_latents` to `num_labels`.\n\nThis was the idea of the original Perceiver paper. However, it could only output classification logits. In a follow-up\nwork, PerceiverIO, they generalized it to let the model also produce outputs of arbitrary size. How, you might ask? The\nidea is actually relatively simple: one defines outputs of an arbitrary size, and then applies cross-attention with the\nlast hidden states of the latents, using the outputs as queries, and the latents as keys and values.\n\nSo let's say one wants to perform masked language modeling (BERT-style) with the Perceiver. As the Perceiver's input\nlength will not have an impact on the computation time of the self-attention layers, one can provide raw bytes,\nproviding `inputs` of length 2048 to the model. If one now masks out certain of these 2048 tokens, one can define the\n`outputs` as being of shape: `(batch_size, 2048, 768)`. Next, one performs cross-attention with the final hidden states\nof the latents to update the `outputs` tensor. After cross-attention, one still has a tensor of shape `(batch_size,\n2048, 768)`. One can then place a regular language modeling head on top, to project the last dimension to the\nvocabulary size of the model, i.e. creating logits of shape `(batch_size, 2048, 262)` (as Perceiver uses a vocabulary\nsize of 262 byte IDs).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perceiver_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Perceiver IO architecture. Taken from the <a href=\"https://arxiv.org/abs/2105.15203\">original paper</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/deepmind/deepmind-research/tree/master/perceiver).\n\n<Tip warning={true}>\n\nPerceiver does **not** work with `torch.nn.DataParallel` due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035)\n\n</Tip>\n\n## Resources\n\n- The quickest way to get started with the Perceiver is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Perceiver).\n- Refer to the [blog post](https://huggingface.co/blog/perceiver) if you want to fully understand how the model works and\nis implemented in the library. Note that the models available in the library only showcase some examples of what you can do\nwith the Perceiver. There are many more use cases, including question answering, named-entity recognition, object detection,\naudio classification, video classification, etc."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 81130b3e-0143-4aa9-ab6a-e827e08baad2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/perceiver/", "title": "Perceiver", "tokens": 2357, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"81130b3e-0143-4aa9-ab6a-e827e08baad2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5804f68f-131a-44e4-b27a-214bc37d2455\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ee7a2699c6d096f65e9c99af40d88a286baf4ff7fcc28d6039fdbc13442c950c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"31215f69-6570-4011-83c2-2c56a5abea24\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver/\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5d15c08b0ee5bed93bd53e5ba446b64b63817023f34eada9fcd1a019adcf6697\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4623, \"end_char_idx\": 9435, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "doc_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "ref_doc_id": "5804f68f-131a-44e4-b27a-214bc37d2455", "chroma:document": "After cross-attention, one still has a tensor of shape `(batch_size,\n2048, 768)`. One can then place a regular language modeling head on top, to project the last dimension to the\nvocabulary size of the model, i.e. creating logits of shape `(batch_size, 2048, 262)` (as Perceiver uses a vocabulary\nsize of 262 byte IDs).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perceiver_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Perceiver IO architecture. Taken from the <a href=\"https://arxiv.org/abs/2105.15203\">original paper</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/deepmind/deepmind-research/tree/master/perceiver).\n\n<Tip warning={true}>\n\nPerceiver does **not** work with `torch.nn.DataParallel` due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035)\n\n</Tip>\n\n## Resources\n\n- The quickest way to get started with the Perceiver is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Perceiver).\n- Refer to the [blog post](https://huggingface.co/blog/perceiver) if you want to fully understand how the model works and\nis implemented in the library. Note that the models available in the library only showcase some examples of what you can do\nwith the Perceiver. There are many more use cases, including question answering, named-entity recognition, object detection,\naudio classification, video classification, etc.\n- [Text classification task guide](../tasks/sequence_classification)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Image classification task guide](../tasks/image_classification)\n\n## Perceiver specific outputs\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverModelOutput\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverDecoderOutput\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverMaskedLMOutput\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverClassifierOutput\n\n## PerceiverConfig\n\n[[autodoc]] PerceiverConfig\n\n## PerceiverTokenizer\n\n[[autodoc]] PerceiverTokenizer\n    - __call__\n\n## PerceiverFeatureExtractor\n\n[[autodoc]] PerceiverFeatureExtractor\n    - __call__\n\n## PerceiverImageProcessor\n\n[[autodoc]] PerceiverImageProcessor\n    - preprocess\n\n## PerceiverTextPreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverTextPreprocessor\n\n## PerceiverImagePreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverImagePreprocessor\n\n## PerceiverOneHotPreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverOneHotPreprocessor\n\n## PerceiverAudioPreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverAudioPreprocessor\n\n## PerceiverMultimodalPreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverMultimodalPreprocessor\n\n## PerceiverProjectionDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverProjectionDecoder\n\n## PerceiverBasicDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverBasicDecoder\n\n## PerceiverClassificationDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverClassificationDecoder\n\n## PerceiverOpticalFlowDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverOpticalFlowDecoder\n\n## PerceiverBasicVideoAutoencodingDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverBasicVideoAutoencodingDecoder\n\n## PerceiverMultimodalDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverMultimodalDecoder\n\n## PerceiverProjectionPostprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverProjectionPostprocessor\n\n## PerceiverAudioPostprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverAudioPostprocessor\n\n## PerceiverClassificationPostprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverClassificationPostprocessor\n\n## PerceiverMultimodalPostprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverMultimodalPostprocessor\n\n## PerceiverModel\n\n[[autodoc]] PerceiverModel\n    - forward\n\n## PerceiverForMaskedLM\n\n[[autodoc]] PerceiverForMaskedLM\n    - forward\n\n## PerceiverForSequenceClassification\n\n[[autodoc]] PerceiverForSequenceClassification\n    - forward\n\n## PerceiverForImageClassificationLearned\n\n[[autodoc]] PerceiverForImageClassificationLearned\n    - forward\n\n## PerceiverForImageClassificationFourier\n\n[[autodoc]] PerceiverForImageClassificationFourier\n    - forward\n\n## PerceiverForImageClassificationConvProcessing\n\n[[autodoc]] PerceiverForImageClassificationConvProcessing\n    - forward\n\n## PerceiverForOpticalFlow\n\n[[autodoc]] PerceiverForOpticalFlow\n    - forward\n\n## PerceiverForMultimodalAutoencoding\n\n[[autodoc]] PerceiverForMultimodalAutoencoding\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 57e1d63d-1783-44c5-a75f-5c642d20ada4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xmod/", "title": "X-MOD", "tokens": 1257, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"57e1d63d-1783-44c5-a75f-5c642d20ada4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod/\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e70c5cc8-476e-4e75-8f8d-affc5b381676\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod/\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6a5a7a90909e908f1d834aa0b854891aaa857102f257bba30558face6cb8068d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"549dd530-59a4-44ca-b4c2-8c691512b8f0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bff30ca492a236fc0a5849542af0e64ee2481ff8dfe39d3cdca96f6e6d1f4f65\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3012, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e70c5cc8-476e-4e75-8f8d-affc5b381676", "doc_id": "e70c5cc8-476e-4e75-8f8d-affc5b381676", "ref_doc_id": "e70c5cc8-476e-4e75-8f8d-affc5b381676", "chroma:document": "# X-MOD\n\n## Overview\n\nThe X-MOD model was proposed in [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](http://dx.doi.org/10.18653/v1/2022.naacl-main.255) by Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe.\nX-MOD extends multilingual masked language models like [XLM-R](xlm-roberta) to include language-specific modular components (_language adapters_) during pre-training. For fine-tuning, the language adapters in each transformer layer are frozen.\n\nThe abstract from the paper is the following:\n\n*Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-MOD) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.*\n\nThis model was contributed by [jvamvas](https://huggingface.co/jvamvas).\nThe original code can be found [here](https://github.com/facebookresearch/fairseq/tree/58cc6cca18f15e6d56e3f60c959fe4f878960a60/fairseq/models/xmod) and the original documentation is found [here](https://github.com/facebookresearch/fairseq/tree/58cc6cca18f15e6d56e3f60c959fe4f878960a60/examples/xmod).\n\n## Usage tips\n\nTips:\n- X-MOD is similar to [XLM-R](xlm-roberta), but a difference is that the input language needs to be specified so that the correct language adapter can be activated.\n- The main models \u2013 base and large \u2013 have adapters for 81 languages.\n\n## Adapter Usage\n\n### Input language\n\nThere are two ways to specify the input language:\n1. By setting a default language before using the model:\n\n```python\nfrom transformers import XmodModel\n\nmodel = XmodModel.from_pretrained(\"facebook/xmod-base\")\nmodel.set_default_language(\"en_XX\")\n```\n\n2. By explicitly passing the index of the language adapter for each sample:\n\n```python\nimport torch\n\ninput_ids = torch.tensor(\n    [\n        [0, 581, 10269, 83, 99942, 136, 60742, 23, 70, 80583, 18276, 2],\n        [0, 1310, 49083, 443, 269, 71, 5486, 165, 60429, 660, 23, 2],\n    ]\n)\nlang_ids = torch.LongTensor(\n    [\n        0,  # en_XX\n        8,  # de_DE\n    ]\n)\noutput = model(input_ids, lang_ids=lang_ids)\n```\n\n### Fine-tuning\nThe paper recommends that the embedding layer and the language adapters are frozen during fine-tuning."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 549dd530-59a4-44ca-b4c2-8c691512b8f0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xmod/", "title": "X-MOD", "tokens": 1257, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"549dd530-59a4-44ca-b4c2-8c691512b8f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod/\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e70c5cc8-476e-4e75-8f8d-affc5b381676\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod/\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6a5a7a90909e908f1d834aa0b854891aaa857102f257bba30558face6cb8068d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"57e1d63d-1783-44c5-a75f-5c642d20ada4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod/\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d2daaa4e8a568feca1309c0bdfcdd802a58e67d1536c83ca102963636161ad3c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1932, \"end_char_idx\": 4348, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e70c5cc8-476e-4e75-8f8d-affc5b381676", "doc_id": "e70c5cc8-476e-4e75-8f8d-affc5b381676", "ref_doc_id": "e70c5cc8-476e-4e75-8f8d-affc5b381676", "chroma:document": "## Usage tips\n\nTips:\n- X-MOD is similar to [XLM-R](xlm-roberta), but a difference is that the input language needs to be specified so that the correct language adapter can be activated.\n- The main models \u2013 base and large \u2013 have adapters for 81 languages.\n\n## Adapter Usage\n\n### Input language\n\nThere are two ways to specify the input language:\n1. By setting a default language before using the model:\n\n```python\nfrom transformers import XmodModel\n\nmodel = XmodModel.from_pretrained(\"facebook/xmod-base\")\nmodel.set_default_language(\"en_XX\")\n```\n\n2. By explicitly passing the index of the language adapter for each sample:\n\n```python\nimport torch\n\ninput_ids = torch.tensor(\n    [\n        [0, 581, 10269, 83, 99942, 136, 60742, 23, 70, 80583, 18276, 2],\n        [0, 1310, 49083, 443, 269, 71, 5486, 165, 60429, 660, 23, 2],\n    ]\n)\nlang_ids = torch.LongTensor(\n    [\n        0,  # en_XX\n        8,  # de_DE\n    ]\n)\noutput = model(input_ids, lang_ids=lang_ids)\n```\n\n### Fine-tuning\nThe paper recommends that the embedding layer and the language adapters are frozen during fine-tuning. A method for doing this is provided:\n\n```python\nmodel.freeze_embeddings_and_language_adapters()\n# Fine-tune the model ...\n```\n\n### Cross-lingual transfer\nAfter fine-tuning, zero-shot cross-lingual transfer can be tested by activating the language adapter of the target language:\n\n```python\nmodel.set_default_language(\"de_DE\")\n# Evaluate the model on German examples ...\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## XmodConfig\n\n[[autodoc]] XmodConfig\n\n## XmodModel\n\n[[autodoc]] XmodModel\n    - forward\n\n## XmodForCausalLM\n\n[[autodoc]] XmodForCausalLM\n    - forward\n\n## XmodForMaskedLM\n\n[[autodoc]] XmodForMaskedLM\n    - forward\n\n## XmodForSequenceClassification\n\n[[autodoc]] XmodForSequenceClassification\n    - forward\n\n## XmodForMultipleChoice\n\n[[autodoc]] XmodForMultipleChoice\n    - forward\n\n## XmodForTokenClassification\n\n[[autodoc]] XmodForTokenClassification\n    - forward\n\n## XmodForQuestionAnswering\n\n[[autodoc]] XmodForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 437b05fb-cd5b-4faf-abac-67fc1f267011
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/distilbert/", "title": "DistilBERT", "tokens": 3717, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"437b05fb-cd5b-4faf-abac-67fc1f267011\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert/\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f973262e-c2e6-4d27-b492-e1523422ec90\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert/\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ea096d62a57ab3f7b46d7be0eb5a768ad9559307cb90040b903dc6806bb92c50\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a33f10fc-b8d0-4de9-8b4a-bbb69542d4db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"04dcd922dca7a9bee23ceed40094064c386afeb924801d6aa7f5d63299c50d9a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3097, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f973262e-c2e6-4d27-b492-e1523422ec90", "doc_id": "f973262e-c2e6-4d27-b492-e1523422ec90", "ref_doc_id": "f973262e-c2e6-4d27-b492-e1523422ec90", "chroma:document": "# DistilBERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=distilbert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-distilbert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/distilbert-base-uncased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n<a href=\"https://huggingface.co/papers/1910.01108\">\n<img alt=\"Paper page\" src=\"https://img.shields.io/badge/Paper%20page-1910.01108-green\">\n</a>\n</div>\n\n## Overview\n\nThe DistilBERT model was proposed in the blog post [Smaller, faster, cheaper, lighter: Introducing DistilBERT, a\ndistilled version of BERT](https://medium.com/huggingface/distilbert-8cf3380435b5), and the paper [DistilBERT, a\ndistilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108). DistilBERT is a\nsmall, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than\n*google-bert/bert-base-uncased*, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language\nunderstanding benchmark.\n\nThe abstract from the paper is the following:\n\n*As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP),\noperating these large models in on-the-edge and/or under constrained computational training or inference budgets\nremains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation\nmodel, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger\ncounterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage\nknowledge distillation during the pretraining phase and show that it is possible to reduce the size of a BERT model by\n40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive\nbiases learned by larger models during pretraining, we introduce a triple loss combining language modeling,\ndistillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we\ndemonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device\nstudy.*\n\nThis model was contributed by [victorsanh](https://huggingface.co/victorsanh). This model jax version was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation).\n\n## Usage tips\n\n- DistilBERT doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just\n  separate your segments with the separation token `tokenizer.sep_token` (or `[SEP]`).\n- DistilBERT doesn't have options to select the input positions (`position_ids` input). This could be added if\n  necessary though, just let us know if you need this option.\n- Same as BERT but smaller."}


