# embeddings_queue

created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 50d7ceed-a11d-4bf9-86df-39c236274b6a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"50d7ceed-a11d-4bf9-86df-39c236274b6a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm/\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1e0eb0e7-a29c-45eb-bf9d-ddc680960939\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm/\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7d65f87bda1d85b5f7510d98b948f32ec8528c9d507578cdac6cca58ba536b45\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78e43ddd-3213-4268-a5f6-ccb808e3cf43\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm/\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"be875441324158dd07e4336889ae7fec948a6e9dd6554e66ec17f800071e45d0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"424e32cc-d6f4-47a8-bb30-1fd361c987a9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fa43d8f6effcc0541c60b7e2482ddef4cc2520021fc3f1ee6d76d3710289f825\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1887, \"end_char_idx\": 5415, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "doc_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "ref_doc_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "chroma:document": "The abstract from \n\"Biological structure and function emerge from scaling unsupervised learning to 250 \nmillion protein sequences\" is\n\n\n*In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised\nlearning has led to major advances in representation learning and statistical generation. In the life sciences, the\nanticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling\nat the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To\nthis end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250\nmillion protein sequences spanning evolutionary diversity. The resulting model contains information about biological\nproperties in its representations. The representations are learned from sequence data alone. The learned representation\nspace has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to\nremote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and\ncan be identified by linear projections. Representation learning produces features that generalize across a range of\napplications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and\nimproving state-of-the-art features for long-range contact prediction.*\n\n\nThe abstract from\n\"Language models of protein sequences at the scale of evolution enable accurate structure prediction\" is\n\n*Large language models have recently been shown to develop emergent capabilities with scale, going beyond\nsimple pattern matching to perform higher level reasoning and generate lifelike images and text. While\nlanguage models trained on protein sequences have been studied at a smaller scale, little is known about\nwhat they learn about biology as they are scaled up. In this work we train models up to 15 billion parameters,\nthe largest language models of proteins to be evaluated to date. We find that as models are scaled they learn\ninformation enabling the prediction of the three-dimensional structure of a protein at the resolution of\nindividual atoms. We present ESMFold for high accuracy end-to-end atomic level structure prediction directly\nfrom the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for\nsequences with low perplexity that are well understood by the language model. ESMFold inference is an\norder of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic\nproteins in practical timescales.*\n\nThe original code can be found [here](https://github.com/facebookresearch/esm) and was\nwas developed by the Fundamental AI Research team at Meta AI.\nESM-1b, ESM-1v and ESM-2 were contributed to huggingface by [jasonliu](https://huggingface.co/jasonliu)\nand [Matt](https://huggingface.co/Rocketknight1).\n\nESMFold was contributed to huggingface by [Matt](https://huggingface.co/Rocketknight1) and\n[Sylvain](https://huggingface.co/sgugger), with a big thank you to Nikita Smetanin, Roshan Rao and Tom Sercu for their\nhelp throughout the process!\n\n## Usage tips\n\n- ESM models are trained with a masked language modeling (MLM) objective.\n- The HuggingFace port of ESMFold uses portions of the [openfold](https://github.com/aqlaboratory/openfold) library. The `openfold` library is licensed under the Apache License 2.0."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 424e32cc-d6f4-47a8-bb30-1fd361c987a9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"424e32cc-d6f4-47a8-bb30-1fd361c987a9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm/\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1e0eb0e7-a29c-45eb-bf9d-ddc680960939\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm/\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7d65f87bda1d85b5f7510d98b948f32ec8528c9d507578cdac6cca58ba536b45\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50d7ceed-a11d-4bf9-86df-39c236274b6a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm/\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4b20c6a3ecfce7789566b0d16d68dd532e97c4c67b2e97eb6ef50511d1065ce5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3872, \"end_char_idx\": 6570, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "doc_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "ref_doc_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "chroma:document": "In this work we train models up to 15 billion parameters,\nthe largest language models of proteins to be evaluated to date. We find that as models are scaled they learn\ninformation enabling the prediction of the three-dimensional structure of a protein at the resolution of\nindividual atoms. We present ESMFold for high accuracy end-to-end atomic level structure prediction directly\nfrom the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for\nsequences with low perplexity that are well understood by the language model. ESMFold inference is an\norder of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic\nproteins in practical timescales.*\n\nThe original code can be found [here](https://github.com/facebookresearch/esm) and was\nwas developed by the Fundamental AI Research team at Meta AI.\nESM-1b, ESM-1v and ESM-2 were contributed to huggingface by [jasonliu](https://huggingface.co/jasonliu)\nand [Matt](https://huggingface.co/Rocketknight1).\n\nESMFold was contributed to huggingface by [Matt](https://huggingface.co/Rocketknight1) and\n[Sylvain](https://huggingface.co/sgugger), with a big thank you to Nikita Smetanin, Roshan Rao and Tom Sercu for their\nhelp throughout the process!\n\n## Usage tips\n\n- ESM models are trained with a masked language modeling (MLM) objective.\n- The HuggingFace port of ESMFold uses portions of the [openfold](https://github.com/aqlaboratory/openfold) library. The `openfold` library is licensed under the Apache License 2.0.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n## EsmConfig\n\n[[autodoc]] EsmConfig\n    - all\n\n## EsmTokenizer\n\n[[autodoc]] EsmTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## EsmModel\n\n[[autodoc]] EsmModel\n    - forward\n\n## EsmForMaskedLM\n\n[[autodoc]] EsmForMaskedLM\n    - forward\n\n## EsmForSequenceClassification\n\n[[autodoc]] EsmForSequenceClassification\n    - forward\n\n## EsmForTokenClassification\n\n[[autodoc]] EsmForTokenClassification\n    - forward\n\n## EsmForProteinFolding\n\n[[autodoc]] EsmForProteinFolding\n    - forward\n\n</pt>\n<tf>\n\n## TFEsmModel\n\n[[autodoc]] TFEsmModel\n    - call\n\n## TFEsmForMaskedLM\n\n[[autodoc]] TFEsmForMaskedLM\n    - call\n\n## TFEsmForSequenceClassification\n\n[[autodoc]] TFEsmForSequenceClassification\n    - call\n\n## TFEsmForTokenClassification\n\n[[autodoc]] TFEsmForTokenClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5d1a149d-f7b4-46ce-bb9d-ec2be55e53b3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt/", "title": "Pyramid Vision Transformer (PVT)", "tokens": 892, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5d1a149d-f7b4-46ce-bb9d-ec2be55e53b3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt/\", \"title\": \"Pyramid Vision Transformer (PVT)\", \"tokens\": 892, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f5cc2cc-1b85-487b-9923-99e86f38f3fe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt/\", \"title\": \"Pyramid Vision Transformer (PVT)\", \"tokens\": 892, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6beaae0361dffd95162d536c3d702cc441189a1b0ffe0a851c700d35f2b3fa43\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3159, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f5cc2cc-1b85-487b-9923-99e86f38f3fe", "doc_id": "9f5cc2cc-1b85-487b-9923-99e86f38f3fe", "ref_doc_id": "9f5cc2cc-1b85-487b-9923-99e86f38f3fe", "chroma:document": "# Pyramid Vision Transformer (PVT)\n\n## Overview\n\nThe PVT model was proposed in\n[Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://arxiv.org/abs/2102.12122)\nby Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao. The PVT is a type of\nvision transformer that utilizes a pyramid structure to make it an effective backbone for dense prediction tasks. Specifically\nit allows for more fine-grained inputs (4 x 4 pixels per patch) to be used, while simultaneously shrinking the sequence length\nof the Transformer as it deepens - reducing the computational cost. Additionally, a spatial-reduction attention (SRA) layer\nis used to further reduce the resource consumption when learning high-resolution features.\n\nThe abstract from the paper is the following:\n\n*Although convolutional neural networks (CNNs) have achieved great success in computer vision, this work investigates a \nsimpler, convolution-free backbone network useful for many dense prediction tasks. Unlike the recently proposed Vision \nTransformer (ViT) that was designed for image classification specifically, we introduce the Pyramid Vision Transformer \n(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several \nmerits compared to current state of the arts. Different from ViT that typically yields low resolution outputs and \nincurs high computational and memory costs, PVT not only can be trained on dense partitions of an image to achieve high \noutput resolution, which is important for dense prediction, but also uses a progressive shrinking pyramid to reduce the \ncomputations of large feature maps. PVT inherits the advantages of both CNN and Transformer, making it a unified \nbackbone for various vision tasks without convolutions, where it can be used as a direct replacement for CNN backbones. \nWe validate PVT through extensive experiments, showing that it boosts the performance of many downstream tasks, including\nobject detection, instance and semantic segmentation. For example, with a comparable number of parameters, PVT+RetinaNet \nachieves 40.4 AP on the COCO dataset, surpassing ResNet50+RetinNet (36.3 AP) by 4.1 absolute AP (see Figure 2). We hope \nthat PVT could serve as an alternative and useful backbone for pixel-level predictions and facilitate future research.*\n\nThis model was contributed by [Xrenya](https://huggingface.co/Xrenya). The original code can be found [here](https://github.com/whai362/PVT).\n\n\n- PVTv1 on ImageNet-1K\n\n| **Model variant**  |**Size** |**Acc@1**|**Params (M)**|\n|--------------------|:-------:|:-------:|:------------:|\n| PVT-Tiny           |    224  |   75.1  |     13.2     |\n| PVT-Small          |    224  |   79.8  |     24.5     |\n| PVT-Medium         |    224  |   81.2  |     44.2     |\n| PVT-Large          |    224  |   81.7  |     61.4     |\n\n\n## PvtConfig\n\n[[autodoc]] PvtConfig\n\n## PvtImageProcessor\n\n[[autodoc]] PvtImageProcessor\n    - preprocess\n\n## PvtForImageClassification\n\n[[autodoc]] PvtForImageClassification\n    - forward\n\n## PvtModel\n\n[[autodoc]] PvtModel\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8311b772-6d45-4431-88e9-f67d1674d72a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8311b772-6d45-4431-88e9-f67d1674d72a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4ed314aa-ddb6-4e35-a41c-d727ec9a3df0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"98a10bd3-13e6-41a2-8591-c8c2652170e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4f73b6a6ca8e6f396f61c8ee559a3b59dcc31c8f41adc78ed9c513735185d76\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3102, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "ref_doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "chroma:document": "# Reformer\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=reformer\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-reformer-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/reformer-crime-and-punishment\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Reformer model was proposed in the paper [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451.pdf) by Nikita Kitaev, \u0141ukasz Kaiser, Anselm Levskaya.\n\nThe abstract from the paper is the following:\n\n*Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can\nbe prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of\nTransformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its\ncomplexity from O(L^2) to O(Llog(L)), where L is the length of the sequence. Furthermore, we use reversible residual\nlayers instead of the standard residuals, which allows storing activations only once in the training process instead of\nN times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models\nwhile being much more memory-efficient and much faster on long sequences.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be\nfound [here](https://github.com/google/trax/tree/master/trax/models/reformer).\n\n## Usage tips\n\n- Reformer does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035).\n- Use Axial position encoding (see below for more details). It\u2019s a mechanism to avoid having a huge positional encoding matrix (when the sequence length is very big) by factorizing it into smaller matrices.\n- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It\u2019s a technique to avoid computing the full product query-key in the attention layers.\n- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).\n- Compute the feedforward operations by chunks and not on the whole batch.\n\n### Axial Positional Encodings\n\nAxial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)\nand developed by the authors of this model's paper. In models that are treating very long input sequences, the\nconventional position id encodings store an embeddings vector of size \\\\(d\\\\) being the `config.hidden_size` for\nevery position \\\\(i, \\ldots, n_s\\\\), with \\\\(n_s\\\\) being `config.max_embedding_size`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 98a10bd3-13e6-41a2-8591-c8c2652170e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"98a10bd3-13e6-41a2-8591-c8c2652170e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4ed314aa-ddb6-4e35-a41c-d727ec9a3df0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8311b772-6d45-4431-88e9-f67d1674d72a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6a7447e86e266685db428201b5966ed90854aeff04340cfe3af238313aeb9e35\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dab8ebd5-ee9f-446f-bf4e-6aa57e5cc967\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0f69baf10066f574a8b8d309e1fb789ea227ec2930c8111245a30fb1f3c43cc7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1493, \"end_char_idx\": 3435, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "ref_doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "chroma:document": "The Authors' code can be\nfound [here](https://github.com/google/trax/tree/master/trax/models/reformer).\n\n## Usage tips\n\n- Reformer does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035).\n- Use Axial position encoding (see below for more details). It\u2019s a mechanism to avoid having a huge positional encoding matrix (when the sequence length is very big) by factorizing it into smaller matrices.\n- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It\u2019s a technique to avoid computing the full product query-key in the attention layers.\n- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).\n- Compute the feedforward operations by chunks and not on the whole batch.\n\n### Axial Positional Encodings\n\nAxial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)\nand developed by the authors of this model's paper. In models that are treating very long input sequences, the\nconventional position id encodings store an embeddings vector of size \\\\(d\\\\) being the `config.hidden_size` for\nevery position \\\\(i, \\ldots, n_s\\\\), with \\\\(n_s\\\\) being `config.max_embedding_size`. This means that having\na sequence length of \\\\(n_s = 2^{19} \\approx 0.5M\\\\) and a `config.hidden_size` of \\\\(d = 2^{10} \\approx 1000\\\\)\nwould result in a position encoding matrix:\n\n$$X_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s\\right]$$\n\nwhich alone has over 500M parameters to store."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: dab8ebd5-ee9f-446f-bf4e-6aa57e5cc967
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"dab8ebd5-ee9f-446f-bf4e-6aa57e5cc967\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4ed314aa-ddb6-4e35-a41c-d727ec9a3df0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"98a10bd3-13e6-41a2-8591-c8c2652170e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fba1c48eccae8338d2ba8321ded0b51751c287f31048c6e8ae9efea80c52e289\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b65c7299-2bc9-4593-a5a8-bd3b03008ac3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"37a387a0577b18f4a59a1f963a979ff706be00179cde55bf3c62d3d399429cd7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1972, \"end_char_idx\": 4393, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "ref_doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "chroma:document": "- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It\u2019s a technique to avoid computing the full product query-key in the attention layers.\n- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).\n- Compute the feedforward operations by chunks and not on the whole batch.\n\n### Axial Positional Encodings\n\nAxial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)\nand developed by the authors of this model's paper. In models that are treating very long input sequences, the\nconventional position id encodings store an embeddings vector of size \\\\(d\\\\) being the `config.hidden_size` for\nevery position \\\\(i, \\ldots, n_s\\\\), with \\\\(n_s\\\\) being `config.max_embedding_size`. This means that having\na sequence length of \\\\(n_s = 2^{19} \\approx 0.5M\\\\) and a `config.hidden_size` of \\\\(d = 2^{10} \\approx 1000\\\\)\nwould result in a position encoding matrix:\n\n$$X_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s\\right]$$\n\nwhich alone has over 500M parameters to store. Axial positional encodings factorize \\\\(X_{i,j}\\\\) into two matrices:\n\n$$X^{1}_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d^1\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s^1\\right]$$\n\nand\n\n$$X^{2}_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d^2\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s^2\\right]$$\n\nwith:\n\n$$d = d^1 + d^2 \\text{ and } n_s = n_s^1 \\times n_s^2 .$$\n\nTherefore the following holds:\n\n$$X_{i,j} = \\begin{cases}\nX^{1}_{i, k}, & \\text{if }\\ i < d^1 \\text{ with } k = j \\mod n_s^1 \\\\\nX^{2}_{i - d^1, l}, & \\text{if } i \\ge d^1 \\text{ with } l = \\lfloor\\frac{j}{n_s^1}\\rfloor\n\\end{cases}$$\n\nIntuitively, this means that a position embedding vector \\\\(x_j \\in \\mathbb{R}^{d}\\\\) is now the composition of two\nfactorized embedding vectors: \\\\(x^1_{k, l} + x^2_{l, k}\\\\), where as the `config.max_embedding_size` dimension\n\\\\(j\\\\) is factorized into \\\\(k \\text{ and } l\\\\). This design ensures that each position embedding vector\n\\\\(x_j\\\\) is unique."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b65c7299-2bc9-4593-a5a8-bd3b03008ac3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b65c7299-2bc9-4593-a5a8-bd3b03008ac3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4ed314aa-ddb6-4e35-a41c-d727ec9a3df0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dab8ebd5-ee9f-446f-bf4e-6aa57e5cc967\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e6e57e7fddda71de401387e91e564729b246ddf7f9e26464662a5893bf3b145e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7366a839-9fbb-45ad-8d4e-f76d988cca48\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab830199aa7d0cc38601c1afdbf9c4b39534517ae9843d8fd974688c695455d1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3436, \"end_char_idx\": 5018, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "ref_doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "chroma:document": "Axial positional encodings factorize \\\\(X_{i,j}\\\\) into two matrices:\n\n$$X^{1}_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d^1\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s^1\\right]$$\n\nand\n\n$$X^{2}_{i,j}, \\text{ with } i \\in \\left[1,\\ldots, d^2\\right] \\text{ and } j \\in \\left[1,\\ldots, n_s^2\\right]$$\n\nwith:\n\n$$d = d^1 + d^2 \\text{ and } n_s = n_s^1 \\times n_s^2 .$$\n\nTherefore the following holds:\n\n$$X_{i,j} = \\begin{cases}\nX^{1}_{i, k}, & \\text{if }\\ i < d^1 \\text{ with } k = j \\mod n_s^1 \\\\\nX^{2}_{i - d^1, l}, & \\text{if } i \\ge d^1 \\text{ with } l = \\lfloor\\frac{j}{n_s^1}\\rfloor\n\\end{cases}$$\n\nIntuitively, this means that a position embedding vector \\\\(x_j \\in \\mathbb{R}^{d}\\\\) is now the composition of two\nfactorized embedding vectors: \\\\(x^1_{k, l} + x^2_{l, k}\\\\), where as the `config.max_embedding_size` dimension\n\\\\(j\\\\) is factorized into \\\\(k \\text{ and } l\\\\). This design ensures that each position embedding vector\n\\\\(x_j\\\\) is unique.\n\nUsing the above example again, axial position encoding with \\\\(d^1 = 2^9, d^2 = 2^9, n_s^1 = 2^9, n_s^2 = 2^{10}\\\\)\ncan drastically reduced the number of parameters from 500 000 000 to \\\\(2^{18} + 2^{19} \\approx 780 000\\\\) parameters, this means 85% less memory usage.\n\nIn practice, the parameter `config.axial_pos_embds_dim` is set to a tuple \\\\((d^1, d^2)\\\\) which sum has to be\nequal to `config.hidden_size` and `config.axial_pos_shape` is set to a tuple \\\\((n_s^1, n_s^2)\\\\) which\nproduct has to be equal to `config.max_embedding_size`, which during training has to be equal to the *sequence\nlength* of the `input_ids`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7366a839-9fbb-45ad-8d4e-f76d988cca48
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7366a839-9fbb-45ad-8d4e-f76d988cca48\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4ed314aa-ddb6-4e35-a41c-d727ec9a3df0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b65c7299-2bc9-4593-a5a8-bd3b03008ac3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9e1d1dd03e40f3503fbb0783f337de2ed711c2f3801a52ef8b5934134b239c88\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"02bcef94-aecd-4870-87cf-2d662e037754\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"396889b2d203e7e020062d3dd57b0db89efcffcab854cdbf59464f8e729464b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4317, \"end_char_idx\": 7468, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "ref_doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "chroma:document": "This design ensures that each position embedding vector\n\\\\(x_j\\\\) is unique.\n\nUsing the above example again, axial position encoding with \\\\(d^1 = 2^9, d^2 = 2^9, n_s^1 = 2^9, n_s^2 = 2^{10}\\\\)\ncan drastically reduced the number of parameters from 500 000 000 to \\\\(2^{18} + 2^{19} \\approx 780 000\\\\) parameters, this means 85% less memory usage.\n\nIn practice, the parameter `config.axial_pos_embds_dim` is set to a tuple \\\\((d^1, d^2)\\\\) which sum has to be\nequal to `config.hidden_size` and `config.axial_pos_shape` is set to a tuple \\\\((n_s^1, n_s^2)\\\\) which\nproduct has to be equal to `config.max_embedding_size`, which during training has to be equal to the *sequence\nlength* of the `input_ids`.\n\n\n### LSH Self Attention\n\nIn Locality sensitive hashing (LSH) self attention the key and query projection weights are tied. Therefore, the key\nquery embedding vectors are also tied. LSH self attention uses the locality sensitive hashing mechanism proposed in\n[Practical and Optimal LSH for Angular Distance](https://arxiv.org/abs/1509.02897) to assign each of the tied key\nquery embedding vectors to one of `config.num_buckets` possible buckets. The premise is that the more \"similar\"\nkey query embedding vectors (in terms of *cosine similarity*) are to each other, the more likely they are assigned to\nthe same bucket.\n\nThe accuracy of the LSH mechanism can be improved by increasing `config.num_hashes` or directly the argument\n`num_hashes` of the forward function so that the output of the LSH self attention better approximates the output\nof the \"normal\" full self attention. The buckets are then sorted and chunked into query key embedding vector chunks\neach of length `config.lsh_chunk_length`. For each chunk, the query embedding vectors attend to its key vectors\n(which are tied to themselves) and to the key embedding vectors of `config.lsh_num_chunks_before` previous\nneighboring chunks and `config.lsh_num_chunks_after` following neighboring chunks.\n\nFor more information, see the [original Paper](https://arxiv.org/abs/2001.04451) or this great [blog post](https://www.pragmatic.ml/reformer-deep-dive/).\n\nNote that `config.num_buckets` can also be factorized into a list \\\\((n_{\\text{buckets}}^1,\nn_{\\text{buckets}}^2)\\\\). This way instead of assigning the query key embedding vectors to one of \\\\((1,\\ldots,\nn_{\\text{buckets}})\\\\) they are assigned to one of \\\\((1-1,\\ldots, n_{\\text{buckets}}^1-1, \\ldots,\n1-n_{\\text{buckets}}^2, \\ldots, n_{\\text{buckets}}^1-n_{\\text{buckets}}^2)\\\\). This is crucial for very long sequences to\nsave memory.\n\nWhen training a model from scratch, it is recommended to leave `config.num_buckets=None`, so that depending on the\nsequence length a good value for `num_buckets` is calculated on the fly. This value will then automatically be\nsaved in the config and should be reused for inference.\n\nUsing LSH self attention, the memory and time complexity of the query-key matmul operation can be reduced from\n\\\\(\\mathcal{O}(n_s \\times n_s)\\\\) to \\\\(\\mathcal{O}(n_s \\times \\log(n_s))\\\\), which usually represents the memory\nand time bottleneck in a transformer model, with \\\\(n_s\\\\) being the sequence length."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 02bcef94-aecd-4870-87cf-2d662e037754
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"02bcef94-aecd-4870-87cf-2d662e037754\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4ed314aa-ddb6-4e35-a41c-d727ec9a3df0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7366a839-9fbb-45ad-8d4e-f76d988cca48\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/reformer/\", \"title\": \"Reformer\", \"tokens\": 2745, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b5644b8499d1ad7868c789a8daa60571a09a3c954084e11541ced90919ce1a7d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7471, \"end_char_idx\": 9746, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "ref_doc_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "chroma:document": "### Local Self Attention\n\nLocal self attention is essentially a \"normal\" self attention layer with key, query and value projections, but is\nchunked so that in each chunk of length `config.local_chunk_length` the query embedding vectors only attends to\nthe key embedding vectors in its chunk and to the key embedding vectors of `config.local_num_chunks_before`\nprevious neighboring chunks and `config.local_num_chunks_after` following neighboring chunks.\n\nUsing Local self attention, the memory and time complexity of the query-key matmul operation can be reduced from\n\\\\(\\mathcal{O}(n_s \\times n_s)\\\\) to \\\\(\\mathcal{O}(n_s \\times \\log(n_s))\\\\), which usually represents the memory\nand time bottleneck in a transformer model, with \\\\(n_s\\\\) being the sequence length.\n\n\n### Training\n\nDuring training, we must ensure that the sequence length is set to a value that can be divided by the least common\nmultiple of `config.lsh_chunk_length` and `config.local_chunk_length` and that the parameters of the Axial\nPositional Encodings are correctly set as described above. Reformer is very memory efficient so that the model can\neasily be trained on sequences as long as 64000 tokens.\n\nFor training, the [`ReformerModelWithLMHead`] should be used as follows:\n\n```python\ninput_ids = tokenizer.encode(\"This is a sentence from the training data\", return_tensors=\"pt\")\nloss = model(input_ids, labels=input_ids)[0]\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n## ReformerConfig\n\n[[autodoc]] ReformerConfig\n\n## ReformerTokenizer\n\n[[autodoc]] ReformerTokenizer\n    - save_vocabulary\n\n## ReformerTokenizerFast\n\n[[autodoc]] ReformerTokenizerFast\n\n## ReformerModel\n\n[[autodoc]] ReformerModel\n    - forward\n\n## ReformerModelWithLMHead\n\n[[autodoc]] ReformerModelWithLMHead\n    - forward\n\n## ReformerForMaskedLM\n\n[[autodoc]] ReformerForMaskedLM\n    - forward\n\n## ReformerForSequenceClassification\n\n[[autodoc]] ReformerForSequenceClassification\n    - forward\n\n## ReformerForQuestionAnswering\n\n[[autodoc]] ReformerForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 38124589-3776-42df-b021-6b73f9ef71d3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/camembert/", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"38124589-3776-42df-b021-6b73f9ef71d3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert/\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1f44f127-133b-4286-8ea0-29818831ad85\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert/\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c2e40960512fb3049555ca0526e4d4259be0943f11e3ed65756e6284e6c969f4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"12782af6-dd35-4994-82bc-aa0462797469\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6fe06c8a56f87437b587d671e99662b382783d5338bc58efe976412b0119f52b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1812, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1f44f127-133b-4286-8ea0-29818831ad85", "doc_id": "1f44f127-133b-4286-8ea0-29818831ad85", "ref_doc_id": "1f44f127-133b-4286-8ea0-29818831ad85", "chroma:document": "# CamemBERT\n\n## Overview\n\nThe CamemBERT model was proposed in [CamemBERT: a Tasty French Language Model](https://arxiv.org/abs/1911.03894) by\n[Louis Martin](https://huggingface.co/louismartin), [Benjamin Muller](https://huggingface.co/benjamin-mlr), [Pedro Javier Ortiz Su\u00e1rez](https://huggingface.co/pjox), Yoann Dupont, Laurent Romary, \u00c9ric Villemonte de la\nClergerie, [Djam\u00e9 Seddah](https://huggingface.co/Djame), and [Beno\u00eet Sagot](https://huggingface.co/sagot). It is based on Facebook's RoBERTa model released in 2019. It is a model\ntrained on 138GB of French text.\n\nThe abstract from the paper is the following:\n\n*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available\nmodels have either been trained on English data or on the concatenation of data in multiple languages. This makes\npractical use of such models --in all languages except English-- very limited. Aiming to address this issue for French,\nwe release CamemBERT, a French version of the Bi-directional Encoders for Transformers (BERT). We measure the\nperformance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging,\ndependency parsing, named-entity recognition, and natural language inference. CamemBERT improves the state of the art\nfor most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and\ndownstream applications for French NLP.*\n\nThis model was contributed by [the ALMAnaCH team (Inria)](https://huggingface.co/almanach). The original code can be found [here](https://camembert-model.fr/).\n\n<Tip>\n\nThis implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well \nas the information relative to the inputs and outputs."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 12782af6-dd35-4994-82bc-aa0462797469
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/camembert/", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"12782af6-dd35-4994-82bc-aa0462797469\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert/\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1f44f127-133b-4286-8ea0-29818831ad85\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert/\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c2e40960512fb3049555ca0526e4d4259be0943f11e3ed65756e6284e6c969f4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"38124589-3776-42df-b021-6b73f9ef71d3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/camembert/\", \"title\": \"CamemBERT\", \"tokens\": 1113, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ddf6810e480d24013f1b84839ac5e493c46c3e2f4c45e1d9d310cd09552b905e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 467, \"end_char_idx\": 3589, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1f44f127-133b-4286-8ea0-29818831ad85", "doc_id": "1f44f127-133b-4286-8ea0-29818831ad85", "ref_doc_id": "1f44f127-133b-4286-8ea0-29818831ad85", "chroma:document": "It is based on Facebook's RoBERTa model released in 2019. It is a model\ntrained on 138GB of French text.\n\nThe abstract from the paper is the following:\n\n*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available\nmodels have either been trained on English data or on the concatenation of data in multiple languages. This makes\npractical use of such models --in all languages except English-- very limited. Aiming to address this issue for French,\nwe release CamemBERT, a French version of the Bi-directional Encoders for Transformers (BERT). We measure the\nperformance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging,\ndependency parsing, named-entity recognition, and natural language inference. CamemBERT improves the state of the art\nfor most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and\ndownstream applications for French NLP.*\n\nThis model was contributed by [the ALMAnaCH team (Inria)](https://huggingface.co/almanach). The original code can be found [here](https://camembert-model.fr/).\n\n<Tip>\n\nThis implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well \nas the information relative to the inputs and outputs.\n\n</Tip>\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## CamembertConfig\n\n[[autodoc]] CamembertConfig\n\n## CamembertTokenizer\n\n[[autodoc]] CamembertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## CamembertTokenizerFast\n\n[[autodoc]] CamembertTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## CamembertModel\n\n[[autodoc]] CamembertModel\n\n## CamembertForCausalLM\n\n[[autodoc]] CamembertForCausalLM\n\n## CamembertForMaskedLM\n\n[[autodoc]] CamembertForMaskedLM\n\n## CamembertForSequenceClassification\n\n[[autodoc]] CamembertForSequenceClassification\n\n## CamembertForMultipleChoice\n\n[[autodoc]] CamembertForMultipleChoice\n\n## CamembertForTokenClassification\n\n[[autodoc]] CamembertForTokenClassification\n\n## CamembertForQuestionAnswering\n\n[[autodoc]] CamembertForQuestionAnswering\n\n</pt>\n<tf>\n\n## TFCamembertModel\n\n[[autodoc]] TFCamembertModel\n\n## TFCamembertForCasualLM\n\n[[autodoc]] TFCamembertForCausalLM\n\n## TFCamembertForMaskedLM\n\n[[autodoc]] TFCamembertForMaskedLM\n\n## TFCamembertForSequenceClassification\n\n[[autodoc]] TFCamembertForSequenceClassification\n\n## TFCamembertForMultipleChoice\n\n[[autodoc]] TFCamembertForMultipleChoice\n\n## TFCamembertForTokenClassification\n\n[[autodoc]] TFCamembertForTokenClassification\n\n## TFCamembertForQuestionAnswering\n\n[[autodoc]] TFCamembertForQuestionAnswering\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7d76109f-258d-4fd5-aa1a-b750981db8a7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese/", "title": "GPT-NeoX-Japanese", "tokens": 867, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7d76109f-258d-4fd5-aa1a-b750981db8a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese/\", \"title\": \"GPT-NeoX-Japanese\", \"tokens\": 867, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c921b33f-8b74-4e3d-abaf-c67a354f4556\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese/\", \"title\": \"GPT-NeoX-Japanese\", \"tokens\": 867, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1f55da0e00641be0817d4a1e6c6b9cf607934c4643f66a10f342dca9d00b28b2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2461, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c921b33f-8b74-4e3d-abaf-c67a354f4556", "doc_id": "c921b33f-8b74-4e3d-abaf-c67a354f4556", "ref_doc_id": "c921b33f-8b74-4e3d-abaf-c67a354f4556", "chroma:document": "# GPT-NeoX-Japanese\n\n## Overview\n\nWe introduce GPT-NeoX-Japanese, which is an autoregressive language model for Japanese, trained on top of [https://github.com/EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).\nJapanese is a unique language with its large vocabulary and a combination of hiragana, katakana, and kanji writing scripts.\nTo address this distinct structure of the Japanese language, we use a [special sub-word tokenizer](https://github.com/tanreinama/Japanese-BPEEncoder_V2). We are very grateful to *tanreinama* for open-sourcing this incredibly helpful tokenizer.\nFollowing the recommendations from Google's research on [PaLM](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), we have removed bias parameters from transformer blocks, achieving better model performance. Please refer [this article](https://medium.com/ml-abeja/training-a-better-gpt-2-93b157662ae4) in detail.\n\nDevelopment of the model was led by [Shinya Otani](https://github.com/SO0529), [Takayoshi Makabe](https://github.com/spider-man-tm), [Anuj Arora](https://github.com/Anuj040), and [Kyo Hattori](https://github.com/go5paopao) from [ABEJA, Inc.](https://www.abejainc.com/). For more information on this model-building activity, please refer [here (ja)](https://tech-blog.abeja.asia/entry/abeja-gpt-project-202207).\n\n### Usage example\n\nThe `generate()` method can be used to generate text using GPT NeoX Japanese model.\n\n```python\n>>> from transformers import GPTNeoXJapaneseForCausalLM, GPTNeoXJapaneseTokenizer\n\n>>> model = GPTNeoXJapaneseForCausalLM.from_pretrained(\"abeja/gpt-neox-japanese-2.7b\")\n>>> tokenizer = GPTNeoXJapaneseTokenizer.from_pretrained(\"abeja/gpt-neox-japanese-2.7b\")\n\n>>> prompt = \"\u4eba\u3068AI\u304c\u5354\u8abf\u3059\u308b\u305f\u3081\u306b\u306f\u3001\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens, skip_special_tokens=True)[0]\n\n>>> print(gen_text)\n\u4eba\u3068AI\u304c\u5354\u8abf\u3059\u308b\u305f\u3081\u306b\u306f\u3001AI\u3068\u4eba\u304c\u5171\u5b58\u3057\u3001AI\u3092\u6b63\u3057\u304f\u7406\u89e3\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\n```\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## GPTNeoXJapaneseConfig\n\n[[autodoc]] GPTNeoXJapaneseConfig\n\n## GPTNeoXJapaneseTokenizer\n\n[[autodoc]] GPTNeoXJapaneseTokenizer\n\n## GPTNeoXJapaneseModel\n\n[[autodoc]] GPTNeoXJapaneseModel\n    - forward\n\n## GPTNeoXJapaneseForCausalLM\n\n[[autodoc]] GPTNeoXJapaneseForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 714f9684-ca93-421f-8fd7-77e268243a92
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mra/", "title": "MRA", "tokens": 618, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"714f9684-ca93-421f-8fd7-77e268243a92\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mra/\", \"title\": \"MRA\", \"tokens\": 618, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb199c94-1e71-4016-a670-f6e70f08b4d5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mra/\", \"title\": \"MRA\", \"tokens\": 618, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ac3499e13f261e5b192296dc634f7b2aa5c4c56cef782f9348b0c29f315218f4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2004, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb199c94-1e71-4016-a670-f6e70f08b4d5", "doc_id": "cb199c94-1e71-4016-a670-f6e70f08b4d5", "ref_doc_id": "cb199c94-1e71-4016-a670-f6e70f08b4d5", "chroma:document": "# MRA\n\n## Overview\n\nThe MRA model was proposed in [Multi Resolution Analysis (MRA) for Approximate Self-Attention](https://arxiv.org/abs/2207.10284) by Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh.\n\nThe abstract from the paper is the following:\n\n*Transformers have emerged as a preferred model for many tasks in natural language processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effective ideas include various prespecified sparsity patterns, low-rank basis expansions and combinations thereof. In this paper, we revisit classical Multiresolution Analysis (MRA) concepts such as Wavelets, whose potential value in this setting remains underexplored thus far. We show that simple approximations based on empirical feedback and design choices informed by modern hardware and implementation challenges, eventually yield a MRA-based approach for self-attention with an excellent performance profile across most criteria of interest. We undertake an extensive set of experiments and demonstrate that this multi-resolution scheme outperforms most efficient self-attention proposals and is favorable for both short and long sequences. Code is available at https://github.com/mlpen/mra-attention.*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03).\nThe original code can be found [here](https://github.com/mlpen/mra-attention).\n\n## MraConfig\n\n[[autodoc]] MraConfig\n\n## MraModel\n\n[[autodoc]] MraModel\n    - forward\n\n## MraForMaskedLM\n\n[[autodoc]] MraForMaskedLM\n    - forward\n\n## MraForSequenceClassification\n\n[[autodoc]] MraForSequenceClassification\n    - forward\n\n## MraForMultipleChoice\n\n[[autodoc]] MraForMultipleChoice\n    - forward\n\n## MraForTokenClassification\n\n[[autodoc]] MraForTokenClassification\n    - forward\n\n## MraForQuestionAnswering\n\n[[autodoc]] MraForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b8a75245-7fe3-4325-88c3-14fa0fd87bdb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b8a75245-7fe3-4325-88c3-14fa0fd87bdb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5fb5af5f-dd72-4c89-be67-84af2736028e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"574e3ebf-3276-442e-895a-a443ca6c2147\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"176a40ea098eb50a28e3baf6d0a48e55146e1f8f1b98c3bf3def86b5d3ff85c0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3109, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "ref_doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "chroma:document": "# Pop2Piano\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/spaces/sweetcocoa/pop2piano\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Pop2Piano model was proposed in [Pop2Piano : Pop Audio-based Piano Cover Generation](https://arxiv.org/abs/2211.00895) by Jongho Choi and Kyogu Lee.\n\nPiano covers of pop music are widely enjoyed, but generating them from music is not a trivial task. It requires great \nexpertise with playing piano as well as knowing different characteristics and melodies of a song. With Pop2Piano you \ncan directly generate a cover from a song's audio waveform. It is the first model to directly generate a piano cover \nfrom pop audio without melody and chord extraction modules. \n\nPop2Piano is an encoder-decoder Transformer model based on [T5](https://arxiv.org/pdf/1910.10683.pdf). The input audio \nis transformed to its waveform and passed to the encoder, which transforms it to a latent representation. The decoder \nuses these latent representations to generate token ids in an autoregressive way. Each token id corresponds to one of four \ndifferent token types: time, velocity, note and 'special'. The token ids are then decoded to their equivalent MIDI file.\n\nThe abstract from the paper is the following:\n\n*Piano covers of pop music are enjoyed by many people. However, the\ntask of automatically generating piano covers of pop music is still\nunderstudied. This is partly due to the lack of synchronized\n{Pop, Piano Cover} data pairs, which made it challenging to apply\nthe latest data-intensive deep learning-based methods. To leverage\nthe power of the data-driven approach, we make a large amount of\npaired and synchronized {Pop, Piano Cover} data using an automated\npipeline. In this paper, we present Pop2Piano, a Transformer network\nthat generates piano covers given waveforms of pop music. To the best\nof our knowledge, this is the first model to generate a piano cover\ndirectly from pop audio without using melody and chord extraction\nmodules. We show that Pop2Piano, trained with our dataset, is capable\nof producing plausible piano covers.*\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\nThe original code can be found [here](https://github.com/sweetcocoa/pop2piano).\n\n## Usage tips\n\n* To use Pop2Piano, you will need to install the \ud83e\udd17 Transformers library, as well as the following third party modules:  \n```bash\npip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy\n```\nPlease note that you may need to restart your runtime after installation.\n* Pop2Piano is an Encoder-Decoder based model like T5.\n* Pop2Piano can be used to generate midi-audio files for a given audio sequence.\n* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.\n* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.\n* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 574e3ebf-3276-442e-895a-a443ca6c2147
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"574e3ebf-3276-442e-895a-a443ca6c2147\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5fb5af5f-dd72-4c89-be67-84af2736028e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b8a75245-7fe3-4325-88c3-14fa0fd87bdb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"daaa5cae7e6631303a13c04b8cf402a3746450216cdc623e2bef542ea6371a70\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3ce389f2-0600-43ad-a0c9-93bc67ad7701\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fecc9aabea6e4a742ba500b56d7e9862610312e364e7698c0b7d6adf18eca638\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1489, \"end_char_idx\": 4222, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "ref_doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "chroma:document": "This is partly due to the lack of synchronized\n{Pop, Piano Cover} data pairs, which made it challenging to apply\nthe latest data-intensive deep learning-based methods. To leverage\nthe power of the data-driven approach, we make a large amount of\npaired and synchronized {Pop, Piano Cover} data using an automated\npipeline. In this paper, we present Pop2Piano, a Transformer network\nthat generates piano covers given waveforms of pop music. To the best\nof our knowledge, this is the first model to generate a piano cover\ndirectly from pop audio without using melody and chord extraction\nmodules. We show that Pop2Piano, trained with our dataset, is capable\nof producing plausible piano covers.*\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\nThe original code can be found [here](https://github.com/sweetcocoa/pop2piano).\n\n## Usage tips\n\n* To use Pop2Piano, you will need to install the \ud83e\udd17 Transformers library, as well as the following third party modules:  \n```bash\npip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy\n```\nPlease note that you may need to restart your runtime after installation.\n* Pop2Piano is an Encoder-Decoder based model like T5.\n* Pop2Piano can be used to generate midi-audio files for a given audio sequence.\n* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.\n* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.\n* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.\n\n## Examples\n\n- Example using HuggingFace Dataset:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> ds = load_dataset(\"sweetcocoa/pop2piano_ci\", split=\"test\")\n\n>>> inputs = processor(\n...     audio=ds[\"audio\"][0][\"array\"], sampling_rate=ds[\"audio\"][0][\"sampling_rate\"], return_tensors=\"pt\"\n... )\n>>> model_output = model.generate(input_features=inputs[\"input_features\"], composer=\"composer1\")\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"][0]\n>>> tokenizer_output.write(\"./Outputs/midi_output.mid\")\n```\n\n- Example using your own audio file:\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> audio, sr = librosa.load(\"<your_audio_file_here>\", sr=44100)  # feel free to change the sr to a suitable value."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3ce389f2-0600-43ad-a0c9-93bc67ad7701
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3ce389f2-0600-43ad-a0c9-93bc67ad7701\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5fb5af5f-dd72-4c89-be67-84af2736028e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"574e3ebf-3276-442e-895a-a443ca6c2147\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1224b1715e5d3b4f0439e5c7635ea2316ded3ee9cb2d46011d22cc5bdf9cec5c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e190ac25-0c43-4572-8ac7-6538dd8e2f02\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"98694f1f8b94235607e3f01d18ce2b306e6bd7554ca8122f8467b919d7bdd31b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2633, \"end_char_idx\": 4983, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "ref_doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "chroma:document": "* Pop2Piano is an Encoder-Decoder based model like T5.\n* Pop2Piano can be used to generate midi-audio files for a given audio sequence.\n* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.\n* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.\n* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.\n\n## Examples\n\n- Example using HuggingFace Dataset:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> ds = load_dataset(\"sweetcocoa/pop2piano_ci\", split=\"test\")\n\n>>> inputs = processor(\n...     audio=ds[\"audio\"][0][\"array\"], sampling_rate=ds[\"audio\"][0][\"sampling_rate\"], return_tensors=\"pt\"\n... )\n>>> model_output = model.generate(input_features=inputs[\"input_features\"], composer=\"composer1\")\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"][0]\n>>> tokenizer_output.write(\"./Outputs/midi_output.mid\")\n```\n\n- Example using your own audio file:\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> audio, sr = librosa.load(\"<your_audio_file_here>\", sr=44100)  # feel free to change the sr to a suitable value.\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = processor(audio=audio, sampling_rate=sr, return_tensors=\"pt\")\n>>> model_output = model.generate(input_features=inputs[\"input_features\"], composer=\"composer1\")\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"][0]\n>>> tokenizer_output.write(\"./Outputs/midi_output.mid\")\n```\n\n- Example of processing multiple audio files in batch:\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> # feel free to change the sr to a suitable value."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e190ac25-0c43-4572-8ac7-6538dd8e2f02
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e190ac25-0c43-4572-8ac7-6538dd8e2f02\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5fb5af5f-dd72-4c89-be67-84af2736028e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3ce389f2-0600-43ad-a0c9-93bc67ad7701\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"07f637235d8892095781781d037f44665c6de124549c5cf318f275052382ef8d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ce118c8-b9da-41d0-8257-6f8d002e75de\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d111db0729e3d39edcd4585a8418407c6ab6d2c475215c82eb65fe86b5fc4bbe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4223, \"end_char_idx\": 7426, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "ref_doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "chroma:document": ">>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = processor(audio=audio, sampling_rate=sr, return_tensors=\"pt\")\n>>> model_output = model.generate(input_features=inputs[\"input_features\"], composer=\"composer1\")\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"][0]\n>>> tokenizer_output.write(\"./Outputs/midi_output.mid\")\n```\n\n- Example of processing multiple audio files in batch:\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor\n\n>>> # feel free to change the sr to a suitable value.\n>>> audio1, sr1 = librosa.load(\"<your_first_audio_file_here>\", sr=44100)  \n>>> audio2, sr2 = librosa.load(\"<your_second_audio_file_here>\", sr=44100)\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> processor = Pop2PianoProcessor.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = processor(audio=[audio1, audio2], sampling_rate=[sr1, sr2], return_attention_mask=True, return_tensors=\"pt\")\n>>> # Since we now generating in batch(2 audios) we must pass the attention_mask\n>>> model_output = model.generate(\n...     input_features=inputs[\"input_features\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     composer=\"composer1\",\n... )\n>>> tokenizer_output = processor.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"]\n\n>>> # Since we now have 2 generated MIDI files\n>>> tokenizer_output[0].write(\"./Outputs/midi_output1.mid\")\n>>> tokenizer_output[1].write(\"./Outputs/midi_output2.mid\")\n```\n\n\n- Example of processing multiple audio files in batch (Using `Pop2PianoFeatureExtractor` and `Pop2PianoTokenizer`):\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoFeatureExtractor, Pop2PianoTokenizer\n\n>>> # feel free to change the sr to a suitable value.\n>>> audio1, sr1 = librosa.load(\"<your_first_audio_file_here>\", sr=44100)  \n>>> audio2, sr2 = librosa.load(\"<your_second_audio_file_here>\", sr=44100)\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> feature_extractor = Pop2PianoFeatureExtractor.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> tokenizer = Pop2PianoTokenizer.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = feature_extractor(\n...     audio=[audio1, audio2], \n...     sampling_rate=[sr1, sr2], \n...     return_attention_mask=True, \n...     return_tensors=\"pt\",\n... )\n>>> # Since we now generating in batch(2 audios) we must pass the attention_mask\n>>> model_output = model.generate(\n...     input_features=inputs[\"input_features\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     composer=\"composer1\",\n... )\n>>> tokenizer_output = tokenizer.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"]\n\n>>> # Since we now have 2 generated MIDI files\n>>> tokenizer_output[0].write(\"./Outputs/midi_output1.mid\")\n>>> tokenizer_output[1].write(\"./Outputs/midi_output2.mid\")\n```"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1ce118c8-b9da-41d0-8257-6f8d002e75de
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1ce118c8-b9da-41d0-8257-6f8d002e75de\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5fb5af5f-dd72-4c89-be67-84af2736028e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e190ac25-0c43-4572-8ac7-6538dd8e2f02\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pop2piano/\", \"title\": \"Pop2Piano\", \"tokens\": 2027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0846bcdfcc20bea26d92e177381a011790c5e80f3eb660a11b14ec8796859ef6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5981, \"end_char_idx\": 7814, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "ref_doc_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "chroma:document": "- Example of processing multiple audio files in batch (Using `Pop2PianoFeatureExtractor` and `Pop2PianoTokenizer`):\n\n```python\n>>> import librosa\n>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoFeatureExtractor, Pop2PianoTokenizer\n\n>>> # feel free to change the sr to a suitable value.\n>>> audio1, sr1 = librosa.load(\"<your_first_audio_file_here>\", sr=44100)  \n>>> audio2, sr2 = librosa.load(\"<your_second_audio_file_here>\", sr=44100)\n>>> model = Pop2PianoForConditionalGeneration.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> feature_extractor = Pop2PianoFeatureExtractor.from_pretrained(\"sweetcocoa/pop2piano\")\n>>> tokenizer = Pop2PianoTokenizer.from_pretrained(\"sweetcocoa/pop2piano\")\n\n>>> inputs = feature_extractor(\n...     audio=[audio1, audio2], \n...     sampling_rate=[sr1, sr2], \n...     return_attention_mask=True, \n...     return_tensors=\"pt\",\n... )\n>>> # Since we now generating in batch(2 audios) we must pass the attention_mask\n>>> model_output = model.generate(\n...     input_features=inputs[\"input_features\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     composer=\"composer1\",\n... )\n>>> tokenizer_output = tokenizer.batch_decode(\n...     token_ids=model_output, feature_extractor_output=inputs\n... )[\"pretty_midi_objects\"]\n\n>>> # Since we now have 2 generated MIDI files\n>>> tokenizer_output[0].write(\"./Outputs/midi_output1.mid\")\n>>> tokenizer_output[1].write(\"./Outputs/midi_output2.mid\")\n```\n\n\n## Pop2PianoConfig\n\n[[autodoc]] Pop2PianoConfig\n\n## Pop2PianoFeatureExtractor\n\n[[autodoc]] Pop2PianoFeatureExtractor\n    - __call__\n\n## Pop2PianoForConditionalGeneration\n\n[[autodoc]] Pop2PianoForConditionalGeneration\n    - forward\n    - generate\n\n## Pop2PianoTokenizer\n\n[[autodoc]] Pop2PianoTokenizer\n    - __call__\n\n## Pop2PianoProcessor\n\n[[autodoc]] Pop2PianoProcessor\n    - __call__"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e525289e-f5ba-44b8-9e90-a98f5e9de3ed
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convnextv2/", "title": "ConvNeXt V2", "tokens": 966, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e525289e-f5ba-44b8-9e90-a98f5e9de3ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2/\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a20dd9a-d460-44d9-95ae-89025c7a70d2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2/\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"38f108b559134f09e24ae7931982b4581c92563f9e6e9e8314c686099b900a9b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"047e1723-ca0f-4802-90ae-4967d806accf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"079ce304f51d159f26bd7a0e31e7fde21db4509f17d12205709b7e679b34f79f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3246, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a20dd9a-d460-44d9-95ae-89025c7a70d2", "doc_id": "5a20dd9a-d460-44d9-95ae-89025c7a70d2", "ref_doc_id": "5a20dd9a-d460-44d9-95ae-89025c7a70d2", "chroma:document": "# ConvNeXt V2\n\n## Overview\n\nThe ConvNeXt V2 model was proposed in [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://arxiv.org/abs/2301.00808) by Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, Saining Xie.\nConvNeXt V2 is a pure convolutional model (ConvNet), inspired by the design of Vision Transformers, and a successor of [ConvNeXT](convnext).\n\nThe abstract from the paper is the following:\n\n*Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt, have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked  autoencoders (MAE). However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnextv2_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ConvNeXt V2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2301.00808\">original paper</a>.</small>\n\nThis model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/facebookresearch/ConvNeXt-V2).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ConvNeXt V2.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ConvNextV2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ConvNextV2Config\n\n[[autodoc]] ConvNextV2Config\n\n## ConvNextV2Model\n\n[[autodoc]] ConvNextV2Model\n    - forward\n\n## ConvNextV2ForImageClassification\n\n[[autodoc]] ConvNextV2ForImageClassification\n    - forward\n\n## TFConvNextV2Model\n\n[[autodoc]] TFConvNextV2Model\n    - call"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 047e1723-ca0f-4802-90ae-4967d806accf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convnextv2/", "title": "ConvNeXt V2", "tokens": 966, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"047e1723-ca0f-4802-90ae-4967d806accf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2/\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a20dd9a-d460-44d9-95ae-89025c7a70d2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2/\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"38f108b559134f09e24ae7931982b4581c92563f9e6e9e8314c686099b900a9b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e525289e-f5ba-44b8-9e90-a98f5e9de3ed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnextv2/\", \"title\": \"ConvNeXt V2\", \"tokens\": 966, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"22e9f25bffb1f2ce07243f172206a1b189e6f46bde1ecc79b8e9e4f17fb47cb1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3249, \"end_char_idx\": 3345, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a20dd9a-d460-44d9-95ae-89025c7a70d2", "doc_id": "5a20dd9a-d460-44d9-95ae-89025c7a70d2", "ref_doc_id": "5a20dd9a-d460-44d9-95ae-89025c7a70d2", "chroma:document": "## TFConvNextV2ForImageClassification\n\n[[autodoc]] TFConvNextV2ForImageClassification\n    - call"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c2b03a1c-20b6-4848-873c-6935867617ac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c2b03a1c-20b6-4848-873c-6935867617ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dc855d8a-d686-47e7-8c5f-dd817429d83d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5150e208065b58a2c9c3756a9d5c79d3cca0f3d2dee1f7f52a31e78e77a112d8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3237, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "ref_doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "chroma:document": "# Donut\n\n## Overview\n\nThe Donut model was proposed in [OCR-free Document Understanding Transformer](https://arxiv.org/abs/2111.15664) by\nGeewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, Seunghyun Park.\nDonut consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform document understanding\ntasks such as document image classification, form understanding and visual question answering.\n\nThe abstract from the paper is the following:\n\n*Understanding document images (e.g., invoices) is a core but challenging task since it requires complex functions such as reading text and a holistic understanding of the document. Current Visual Document Understanding (VDU) methods outsource the task of reading text to off-the-shelf Optical Character Recognition (OCR) engines and focus on the understanding task with the OCR outputs. Although such OCR-based approaches have shown promising performance, they suffer from 1) high computational costs for using OCR; 2) inflexibility of OCR models on languages or types of document; 3) OCR error propagation to the subsequent process. To address these issues, in this paper, we introduce a novel OCR-free VDU model named Donut, which stands for Document understanding transformer. As the first step in OCR-free VDU research, we propose a simple architecture (i.e., Transformer) with a pre-training objective (i.e., cross-entropy loss). Donut is conceptually simple yet effective. Through extensive experiments and analyses, we show a simple OCR-free VDU model, Donut, achieves state-of-the-art performances on various VDU tasks in terms of both speed and accuracy. In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/donut_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Donut high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2111.15664\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/clovaai/donut).\n\n## Usage tips\n\n- The quickest way to get started with Donut is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Donut), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- Donut is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.\n\n## Inference examples\n\nDonut's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`DonutImageProcessor`] class is responsible for preprocessing the input image and\n[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`] decodes the generated target tokens to the target string. The\n[`DonutProcessor`] wraps [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]\ninto a single instance to both extract the input features and decode the predicted token ids."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: dc855d8a-d686-47e7-8c5f-dd817429d83d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"dc855d8a-d686-47e7-8c5f-dd817429d83d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c2b03a1c-20b6-4848-873c-6935867617ac\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"aa308d903d3b68159d1953e904373fe7df4d02bba508341f0be3f244dfdd3b28\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"56b1a2f7-629e-4c38-987c-e09a34561221\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ebaa59bff908fade149153d343a38d8556911e98881af4ee450717dd2e82be83\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1715, \"end_char_idx\": 4745, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "ref_doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "chroma:document": "In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/donut_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Donut high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2111.15664\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/clovaai/donut).\n\n## Usage tips\n\n- The quickest way to get started with Donut is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Donut), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- Donut is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.\n\n## Inference examples\n\nDonut's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`DonutImageProcessor`] class is responsible for preprocessing the input image and\n[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`] decodes the generated target tokens to the target string. The\n[`DonutProcessor`] wraps [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step Document Image Classification\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-rvlcdip\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-rvlcdip\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[1][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_rvlcdip>\"\n>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 56b1a2f7-629e-4c38-987c-e09a34561221
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"56b1a2f7-629e-4c38-987c-e09a34561221\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dc855d8a-d686-47e7-8c5f-dd817429d83d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"28fd5347b67de6d67848bf69c3e6f03caed5e2554a447970e924f0b0f9ee3bbd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"21d11092-a4ae-47f4-826e-25e30d29fa9a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0ae56e88d13670c9a9dcff7cad024723ecc5c51f867ae4b1dbfad946f98c5b9e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3239, \"end_char_idx\": 6382, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "ref_doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "chroma:document": "- Step-by-step Document Image Classification\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-rvlcdip\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-rvlcdip\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[1][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_rvlcdip>\"\n>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<.*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'class': 'advertisement'}\n```\n\n- Step-by-step Document Parsing\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-cord-v2\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-cord-v2\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[2][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_cord-v2>\"\n>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 21d11092-a4ae-47f4-826e-25e30d29fa9a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"21d11092-a4ae-47f4-826e-25e30d29fa9a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"56b1a2f7-629e-4c38-987c-e09a34561221\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"53e3926a763c424fdd6db06a4b62f36b8df59a3058959587b56cec7f8e47a1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b2de6d9e-e8f3-4061-8cc0-92c795bcc5ed\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"44d407268a4750aa1975cf042079097ed14e1cf00ef0b8c79213157a6c4e935e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4745, \"end_char_idx\": 7541, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "ref_doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "chroma:document": "*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'class': 'advertisement'}\n```\n\n- Step-by-step Document Parsing\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-cord-v2\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-cord-v2\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[2][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_cord-v2>\"\n>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<.*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'menu': {'nm': 'CINNAMON SUGAR', 'unitprice': '17,000', 'cnt': '1 x', 'price': '17,000'}, 'sub_total': {'subtotal_price': '17,000'}, 'total': {'total_price': '17,000', 'cashprice': '20,000', 'changeprice': '3,000'}}\n```\n\n- Step-by-step Document Visual Question Answering (DocVQA)\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-docvqa\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-docvqa\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image from the DocVQA dataset\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[0][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_docvqa><s_question>{user_input}</s_question><s_answer>\"\n>>> question = \"When is the coffee break?\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b2de6d9e-e8f3-4061-8cc0-92c795bcc5ed
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b2de6d9e-e8f3-4061-8cc0-92c795bcc5ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"21d11092-a4ae-47f4-826e-25e30d29fa9a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/donut/\", \"title\": \"Donut\", \"tokens\": 2384, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7269ccbd517bd838793b4dd149d17dec7d951e21c75602bdecb928aae9aa717f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6382, \"end_char_idx\": 9209, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "ref_doc_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "chroma:document": "*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'menu': {'nm': 'CINNAMON SUGAR', 'unitprice': '17,000', 'cnt': '1 x', 'price': '17,000'}, 'sub_total': {'subtotal_price': '17,000'}, 'total': {'total_price': '17,000', 'cashprice': '20,000', 'changeprice': '3,000'}}\n```\n\n- Step-by-step Document Visual Question Answering (DocVQA)\n\n```py\n>>> import re\n\n>>> from transformers import DonutProcessor, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> processor = DonutProcessor.from_pretrained(\"naver-clova-ix/donut-base-finetuned-docvqa\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"naver-clova-ix/donut-base-finetuned-docvqa\")\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> model.to(device)  # doctest: +IGNORE_RESULT\n\n>>> # load document image from the DocVQA dataset\n>>> dataset = load_dataset(\"hf-internal-testing/example-documents\", split=\"test\")\n>>> image = dataset[0][\"image\"]\n\n>>> # prepare decoder inputs\n>>> task_prompt = \"<s_docvqa><s_question>{user_input}</s_question><s_answer>\"\n>>> question = \"When is the coffee break?\"\n>>> prompt = task_prompt.replace(\"{user_input}\", question)\n>>> decoder_input_ids = processor.tokenizer(prompt, add_special_tokens=False, return_tensors=\"pt\").input_ids\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n\n>>> outputs = model.generate(\n...     pixel_values.to(device),\n...     decoder_input_ids=decoder_input_ids.to(device),\n...     max_length=model.decoder.config.max_position_embeddings,\n...     pad_token_id=processor.tokenizer.pad_token_id,\n...     eos_token_id=processor.tokenizer.eos_token_id,\n...     use_cache=True,\n...     bad_words_ids=[[processor.tokenizer.unk_token_id]],\n...     return_dict_in_generate=True,\n... )\n\n>>> sequence = processor.batch_decode(outputs.sequences)[0]\n>>> sequence = sequence.replace(processor.tokenizer.eos_token, \"\").replace(processor.tokenizer.pad_token, \"\")\n>>> sequence = re.sub(r\"<.*?>\", \"\", sequence, count=1).strip()  # remove first task start token\n>>> print(processor.token2json(sequence))\n{'question': 'When is the coffee break?', 'answer': '11-14 to 11:39 a.m.'}\n```\n\nSee the [model hub](https://huggingface.co/models?filter=donut) to look for Donut checkpoints.\n\n## Training\n\nWe refer to the [tutorial notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Donut).\n\n## DonutSwinConfig\n\n[[autodoc]] DonutSwinConfig\n\n## DonutImageProcessor\n\n[[autodoc]] DonutImageProcessor\n    - preprocess\n\n## DonutFeatureExtractor\n\n[[autodoc]] DonutFeatureExtractor\n    - __call__\n\n## DonutProcessor\n\n[[autodoc]] DonutProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## DonutSwinModel\n\n[[autodoc]] DonutSwinModel\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5804001f-de9b-4d4e-9b33-09b34fb43351
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mluke/", "title": "mLUKE", "tokens": 722, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5804001f-de9b-4d4e-9b33-09b34fb43351\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mluke/\", \"title\": \"mLUKE\", \"tokens\": 722, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e2d1b7db-f527-4928-b7e7-6c0e36afcc3b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mluke/\", \"title\": \"mLUKE\", \"tokens\": 722, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"626d8ad3a7f71f87f96d7ff03817cabc3a903c786e6a001d2c926f31ca207f05\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2471, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e2d1b7db-f527-4928-b7e7-6c0e36afcc3b", "doc_id": "e2d1b7db-f527-4928-b7e7-6c0e36afcc3b", "ref_doc_id": "e2d1b7db-f527-4928-b7e7-6c0e36afcc3b", "chroma:document": "# mLUKE\n\n## Overview\n\nThe mLUKE model was proposed in [mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models](https://arxiv.org/abs/2110.08151) by Ryokan Ri, Ikuya Yamada, and Yoshimasa Tsuruoka. It's a multilingual extension\nof the [LUKE model](https://arxiv.org/abs/2010.01057) trained on the basis of XLM-RoBERTa.\n\nIt is based on XLM-RoBERTa and adds entity embeddings, which helps improve performance on various downstream tasks\ninvolving reasoning about entities such as named entity recognition, extractive question answering, relation\nclassification, cloze-style knowledge completion.\n\nThe abstract from the paper is the following:\n\n*Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual\nalignment information from Wikipedia entities. However, existing methods only exploit entity information in pretraining\nand do not explicitly use entities in downstream tasks. In this study, we explore the effectiveness of leveraging\nentity representations for downstream cross-lingual tasks. We train a multilingual language model with 24 languages\nwith entity representations and show the model consistently outperforms word-based pretrained models in various\ncross-lingual transfer tasks. We also analyze the model and the key insight is that incorporating entity\nrepresentations into the input allows us to extract more language-agnostic features. We also evaluate the model with a\nmultilingual cloze prompt task with the mLAMA dataset. We show that entity-based prompt elicits correct factual\nknowledge more likely than using only word representations.*\n\nThis model was contributed by [ryo0634](https://huggingface.co/ryo0634). The original code can be found [here](https://github.com/studio-ousia/luke).\n\n## Usage tips\n\nOne can directly plug in the weights of mLUKE into a LUKE model, like so:\n\n```python\nfrom transformers import LukeModel\n\nmodel = LukeModel.from_pretrained(\"studio-ousia/mluke-base\")\n```\n\nNote that mLUKE has its own tokenizer, [`MLukeTokenizer`]. You can initialize it as follows:\n\n```python\nfrom transformers import MLukeTokenizer\n\ntokenizer = MLukeTokenizer.from_pretrained(\"studio-ousia/mluke-base\")\n```\n\n<Tip>\n\nAs mLUKE's architecture is equivalent to that of LUKE, one can refer to [LUKE's documentation page](luke) for all\ntips, code examples and notebooks.\n\n</Tip>\n\n## MLukeTokenizer\n\n[[autodoc]] MLukeTokenizer\n    - __call__\n    - save_vocabulary"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: df73df6f-7e52-465e-80c7-914a99ee1535
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"df73df6f-7e52-465e-80c7-914a99ee1535\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c96e7ff9-5fdb-4083-a8ff-bb8f60387115\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0d095170027a7cdff49f30177ff6088b23f5c9d2fb1716f4393582fe8da6d6d6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3ae56397-8753-45f1-abd5-21401d75d964\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d02c7793a966c9e1c22ced7965c51a1d9845be5fcfb0e57cf219e9f028a4fd01\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3038, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "doc_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "ref_doc_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "chroma:document": "# QDQBERT\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe QDQBERT model can be referenced in [Integer Quantization for Deep Learning Inference: Principles and Empirical\nEvaluation](https://arxiv.org/abs/2004.09602) by Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev and Paulius\nMicikevicius.\n\nThe abstract from the paper is the following:\n\n*Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by\ntaking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of\nquantization parameters and evaluate their choices on a wide range of neural network models for different application\ndomains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration\nby processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is\nable to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are\nmore difficult to quantize, such as MobileNets and BERT-large.*\n\nThis model was contributed by [shangz](https://huggingface.co/shangz).\n\n## Usage tips\n\n- QDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to (i) linear layer\n  inputs and weights, (ii) matmul inputs, (iii) residual add inputs, in BERT model.\n- QDQBERT requires the dependency of [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). To install `pip install pytorch-quantization --extra-index-url https://pypi.ngc.nvidia.com`\n- QDQBERT model can be loaded from any checkpoint of HuggingFace BERT model (for example *google-bert/bert-base-uncased*), and\n  perform Quantization Aware Training/Post Training Quantization.\n- A complete example of using QDQBERT model to perform Quatization Aware Training and Post Training Quantization for\n  SQUAD task can be found at [transformers/examples/research_projects/quantization-qdqbert/](examples/research_projects/quantization-qdqbert/).\n\n### Set default quantizers\n\nQDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to BERT by\n`TensorQuantizer` in [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). `TensorQuantizer` is the module\nfor quantizing tensors, with `QuantDescriptor` defining how the tensor should be quantized. Refer to [Pytorch\nQuantization Toolkit userguide](https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/userguide.html) for more details.\n\nBefore creating QDQBERT model, one has to set the default `QuantDescriptor` defining default tensor quantizers."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3ae56397-8753-45f1-abd5-21401d75d964
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3ae56397-8753-45f1-abd5-21401d75d964\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c96e7ff9-5fdb-4083-a8ff-bb8f60387115\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0d095170027a7cdff49f30177ff6088b23f5c9d2fb1716f4393582fe8da6d6d6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"df73df6f-7e52-465e-80c7-914a99ee1535\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5002698f76f6f278b8315fbb01083029a901a5cd0aa89e2fc428c4f7b96b631d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c758f6fe-dded-4129-aa6d-5a6d1602673e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b0f3520e3bc68e6f7e18b8fb72f58f64bebf58b5a7697edc39fa8cdffd565b90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1471, \"end_char_idx\": 4746, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "doc_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "ref_doc_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "chroma:document": "## Usage tips\n\n- QDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to (i) linear layer\n  inputs and weights, (ii) matmul inputs, (iii) residual add inputs, in BERT model.\n- QDQBERT requires the dependency of [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). To install `pip install pytorch-quantization --extra-index-url https://pypi.ngc.nvidia.com`\n- QDQBERT model can be loaded from any checkpoint of HuggingFace BERT model (for example *google-bert/bert-base-uncased*), and\n  perform Quantization Aware Training/Post Training Quantization.\n- A complete example of using QDQBERT model to perform Quatization Aware Training and Post Training Quantization for\n  SQUAD task can be found at [transformers/examples/research_projects/quantization-qdqbert/](examples/research_projects/quantization-qdqbert/).\n\n### Set default quantizers\n\nQDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to BERT by\n`TensorQuantizer` in [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). `TensorQuantizer` is the module\nfor quantizing tensors, with `QuantDescriptor` defining how the tensor should be quantized. Refer to [Pytorch\nQuantization Toolkit userguide](https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/userguide.html) for more details.\n\nBefore creating QDQBERT model, one has to set the default `QuantDescriptor` defining default tensor quantizers.\n\nExample:\n\n```python\n>>> import pytorch_quantization.nn as quant_nn\n>>> from pytorch_quantization.tensor_quant import QuantDescriptor\n\n>>> # The default tensor quantizer is set to use Max calibration method\n>>> input_desc = QuantDescriptor(num_bits=8, calib_method=\"max\")\n>>> # The default tensor quantizer is set to be per-channel quantization for weights\n>>> weight_desc = QuantDescriptor(num_bits=8, axis=((0,)))\n>>> quant_nn.QuantLinear.set_default_quant_desc_input(input_desc)\n>>> quant_nn.QuantLinear.set_default_quant_desc_weight(weight_desc)\n```\n\n### Calibration\n\nCalibration is the terminology of passing data samples to the quantizer and deciding the best scaling factors for\ntensors. After setting up the tensor quantizers, one can use the following example to calibrate the model:\n\n```python\n>>> # Find the TensorQuantizer and enable calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.enable_calib()\n...         module.disable_quant()  # Use full precision data to calibrate\n\n>>> # Feeding data samples\n>>> model(x)\n>>> # ...\n\n>>> # Finalize calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.load_calib_amax()\n...         module.enable_quant()\n\n>>> # If running on GPU, it needs to call .cuda() again because new tensors will be created by calibration process\n>>> model.cuda()\n\n>>> # Keep running the quantized model\n>>> # ...\n```\n\n### Export to ONNX\n\nThe goal of exporting to ONNX is to deploy inference by [TensorRT](https://developer.nvidia.com/tensorrt). Fake\nquantization will be broken into a pair of QuantizeLinear/DequantizeLinear ONNX ops."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c758f6fe-dded-4129-aa6d-5a6d1602673e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c758f6fe-dded-4129-aa6d-5a6d1602673e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c96e7ff9-5fdb-4083-a8ff-bb8f60387115\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0d095170027a7cdff49f30177ff6088b23f5c9d2fb1716f4393582fe8da6d6d6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3ae56397-8753-45f1-abd5-21401d75d964\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d444518b5e7c58447c871653222cc1c56aec21cdc10ca00e760559cd14e6ca58\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1793304a-2547-4b61-908c-d077da41065d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"94d669c0582a3a62ad85cf27e03dbb584c20ba09374ff128c865a1a9938ac725\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3040, \"end_char_idx\": 4973, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "doc_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "ref_doc_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "chroma:document": "Example:\n\n```python\n>>> import pytorch_quantization.nn as quant_nn\n>>> from pytorch_quantization.tensor_quant import QuantDescriptor\n\n>>> # The default tensor quantizer is set to use Max calibration method\n>>> input_desc = QuantDescriptor(num_bits=8, calib_method=\"max\")\n>>> # The default tensor quantizer is set to be per-channel quantization for weights\n>>> weight_desc = QuantDescriptor(num_bits=8, axis=((0,)))\n>>> quant_nn.QuantLinear.set_default_quant_desc_input(input_desc)\n>>> quant_nn.QuantLinear.set_default_quant_desc_weight(weight_desc)\n```\n\n### Calibration\n\nCalibration is the terminology of passing data samples to the quantizer and deciding the best scaling factors for\ntensors. After setting up the tensor quantizers, one can use the following example to calibrate the model:\n\n```python\n>>> # Find the TensorQuantizer and enable calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.enable_calib()\n...         module.disable_quant()  # Use full precision data to calibrate\n\n>>> # Feeding data samples\n>>> model(x)\n>>> # ...\n\n>>> # Finalize calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.load_calib_amax()\n...         module.enable_quant()\n\n>>> # If running on GPU, it needs to call .cuda() again because new tensors will be created by calibration process\n>>> model.cuda()\n\n>>> # Keep running the quantized model\n>>> # ...\n```\n\n### Export to ONNX\n\nThe goal of exporting to ONNX is to deploy inference by [TensorRT](https://developer.nvidia.com/tensorrt). Fake\nquantization will be broken into a pair of QuantizeLinear/DequantizeLinear ONNX ops. After setting static member of\nTensorQuantizer to use Pytorch\u2019s own fake quantization functions, fake quantized model can be exported to ONNX, follow\nthe instructions in [torch.onnx](https://pytorch.org/docs/stable/onnx.html)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1793304a-2547-4b61-908c-d077da41065d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1793304a-2547-4b61-908c-d077da41065d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c96e7ff9-5fdb-4083-a8ff-bb8f60387115\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0d095170027a7cdff49f30177ff6088b23f5c9d2fb1716f4393582fe8da6d6d6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c758f6fe-dded-4129-aa6d-5a6d1602673e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qdqbert/\", \"title\": \"QDQBERT\", \"tokens\": 1678, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2e10e9955d73070cfbd45b8dd439bca0d531226326f34b81f2ebd495b462fc05\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3734, \"end_char_idx\": 6292, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "doc_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "ref_doc_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "chroma:document": "After setting up the tensor quantizers, one can use the following example to calibrate the model:\n\n```python\n>>> # Find the TensorQuantizer and enable calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.enable_calib()\n...         module.disable_quant()  # Use full precision data to calibrate\n\n>>> # Feeding data samples\n>>> model(x)\n>>> # ...\n\n>>> # Finalize calibration\n>>> for name, module in model.named_modules():\n...     if name.endswith(\"_input_quantizer\"):\n...         module.load_calib_amax()\n...         module.enable_quant()\n\n>>> # If running on GPU, it needs to call .cuda() again because new tensors will be created by calibration process\n>>> model.cuda()\n\n>>> # Keep running the quantized model\n>>> # ...\n```\n\n### Export to ONNX\n\nThe goal of exporting to ONNX is to deploy inference by [TensorRT](https://developer.nvidia.com/tensorrt). Fake\nquantization will be broken into a pair of QuantizeLinear/DequantizeLinear ONNX ops. After setting static member of\nTensorQuantizer to use Pytorch\u2019s own fake quantization functions, fake quantized model can be exported to ONNX, follow\nthe instructions in [torch.onnx](https://pytorch.org/docs/stable/onnx.html). Example:\n\n```python\n>>> from pytorch_quantization.nn import TensorQuantizer\n\n>>> TensorQuantizer.use_fb_fake_quant = True\n\n>>> # Load the calibrated model\n>>> ...\n>>> # ONNX export\n>>> torch.onnx.export(...)\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## QDQBertConfig\n\n[[autodoc]] QDQBertConfig\n\n## QDQBertModel\n\n[[autodoc]] QDQBertModel\n    - forward\n\n## QDQBertLMHeadModel\n\n[[autodoc]] QDQBertLMHeadModel\n    - forward\n\n## QDQBertForMaskedLM\n\n[[autodoc]] QDQBertForMaskedLM\n    - forward\n\n## QDQBertForSequenceClassification\n\n[[autodoc]] QDQBertForSequenceClassification\n    - forward\n\n## QDQBertForNextSentencePrediction\n\n[[autodoc]] QDQBertForNextSentencePrediction\n    - forward\n\n## QDQBertForMultipleChoice\n\n[[autodoc]] QDQBertForMultipleChoice\n    - forward\n\n## QDQBertForTokenClassification\n\n[[autodoc]] QDQBertForTokenClassification\n    - forward\n\n## QDQBertForQuestionAnswering\n\n[[autodoc]] QDQBertForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5a0e3a96-a426-4be3-8f4e-ada8d277259f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert-generation/", "title": "BertGeneration", "tokens": 1045, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5a0e3a96-a426-4be3-8f4e-ada8d277259f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation/\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d98f8565-408c-4a39-8983-6e4022852f85\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation/\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1c86cf30c271c0ce0c51e89f86532dedc837b48365ec8370423902f560a9e5e2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"00cb74b5-33d1-4314-95f2-e5507afd8ef7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c58b9b3cc7d9f7c5e20fa00d0f57f2c6cb7d0fcfe6c7023bdd60d970576aaf52\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3380, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d98f8565-408c-4a39-8983-6e4022852f85", "doc_id": "d98f8565-408c-4a39-8983-6e4022852f85", "ref_doc_id": "d98f8565-408c-4a39-8983-6e4022852f85", "chroma:document": "# BertGeneration\n\n## Overview\n\nThe BertGeneration model is a BERT model that can be leveraged for sequence-to-sequence tasks using\n[`EncoderDecoderModel`] as proposed in [Leveraging Pre-trained Checkpoints for Sequence Generation\nTasks](https://arxiv.org/abs/1907.12461) by Sascha Rothe, Shashi Narayan, Aliaksei Severyn.\n\nThe abstract from the paper is the following:\n\n*Unsupervised pretraining of large neural models has recently revolutionized Natural Language Processing. By\nwarm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple\nbenchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language\nUnderstanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We\ndeveloped a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT,\nGPT-2 and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both\nencoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation,\nText Summarization, Sentence Splitting, and Sentence Fusion.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be\nfound [here](https://tfhub.dev/s?module-type=text-generation&subtype=module,placeholder).\n\n## Usage examples and tips\n\nThe model can be used in combination with the [`EncoderDecoderModel`] to leverage two pretrained BERT checkpoints for \nsubsequent fine-tuning:\n\n```python\n>>> # leverage checkpoints for Bert2Bert model...\n>>> # use BERT's cls token as BOS token and sep token as EOS token\n>>> encoder = BertGenerationEncoder.from_pretrained(\"google-bert/bert-large-uncased\", bos_token_id=101, eos_token_id=102)\n>>> # add cross attention layers and use BERT's cls token as BOS token and sep token as EOS token\n>>> decoder = BertGenerationDecoder.from_pretrained(\n...     \"google-bert/bert-large-uncased\", add_cross_attention=True, is_decoder=True, bos_token_id=101, eos_token_id=102\n... )\n>>> bert2bert = EncoderDecoderModel(encoder=encoder, decoder=decoder)\n\n>>> # create tokenizer...\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-large-uncased\")\n\n>>> input_ids = tokenizer(\n...     \"This is a long article to summarize\", add_special_tokens=False, return_tensors=\"pt\"\n... ).input_ids\n>>> labels = tokenizer(\"This is a short summary\", return_tensors=\"pt\").input_ids\n\n>>> # train...\n>>> loss = bert2bert(input_ids=input_ids, decoder_input_ids=labels, labels=labels).loss\n>>> loss.backward()\n```\n\nPretrained [`EncoderDecoderModel`] are also directly available in the model hub, e.g.:\n\n```python\n>>> # instantiate sentence fusion model\n>>> sentence_fuser = EncoderDecoderModel.from_pretrained(\"google/roberta2roberta_L-24_discofuse\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/roberta2roberta_L-24_discofuse\")\n\n>>> input_ids = tokenizer(\n...     \"This is the first sentence. This is the second sentence.\", add_special_tokens=False, return_tensors=\"pt\"\n... ).input_ids\n\n>>> outputs = sentence_fuser.generate(input_ids)\n\n>>> print(tokenizer.decode(outputs[0]))\n```\n\nTips:\n\n- [`BertGenerationEncoder`] and [`BertGenerationDecoder`] should be used in\n  combination with [`EncoderDecoder`]."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 00cb74b5-33d1-4314-95f2-e5507afd8ef7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert-generation/", "title": "BertGeneration", "tokens": 1045, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"00cb74b5-33d1-4314-95f2-e5507afd8ef7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation/\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d98f8565-408c-4a39-8983-6e4022852f85\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation/\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1c86cf30c271c0ce0c51e89f86532dedc837b48365ec8370423902f560a9e5e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5a0e3a96-a426-4be3-8f4e-ada8d277259f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-generation/\", \"title\": \"BertGeneration\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2f9e72c01730cecf737d844d32f53015442bcf6cc224c8862fc7873538d78270\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2767, \"end_char_idx\": 3863, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d98f8565-408c-4a39-8983-6e4022852f85", "doc_id": "d98f8565-408c-4a39-8983-6e4022852f85", "ref_doc_id": "d98f8565-408c-4a39-8983-6e4022852f85", "chroma:document": ":\n\n```python\n>>> # instantiate sentence fusion model\n>>> sentence_fuser = EncoderDecoderModel.from_pretrained(\"google/roberta2roberta_L-24_discofuse\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/roberta2roberta_L-24_discofuse\")\n\n>>> input_ids = tokenizer(\n...     \"This is the first sentence. This is the second sentence.\", add_special_tokens=False, return_tensors=\"pt\"\n... ).input_ids\n\n>>> outputs = sentence_fuser.generate(input_ids)\n\n>>> print(tokenizer.decode(outputs[0]))\n```\n\nTips:\n\n- [`BertGenerationEncoder`] and [`BertGenerationDecoder`] should be used in\n  combination with [`EncoderDecoder`].\n- For summarization, sentence splitting, sentence fusion and translation, no special tokens are required for the input.\n  Therefore, no EOS token should be added to the end of the input.\n\n## BertGenerationConfig\n\n[[autodoc]] BertGenerationConfig\n\n## BertGenerationTokenizer\n\n[[autodoc]] BertGenerationTokenizer\n    - save_vocabulary\n\n## BertGenerationEncoder\n\n[[autodoc]] BertGenerationEncoder\n    - forward\n\n## BertGenerationDecoder\n\n[[autodoc]] BertGenerationDecoder\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b035a8be-30c7-46cc-972e-a342e23c7d39
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b035a8be-30c7-46cc-972e-a342e23c7d39\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl/\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0cf4b87-3f26-4ce2-af82-7f74ab96e191\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl/\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a5c23b428571e98b67a1340548c16bc1e47739fe122bf0a4eaedca03e3a8146e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f8fbb773-0f03-4b16-aea0-2cf4a7901c8a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a5607f6ecfb741e936af0da4910c5c2d5241e33ab33b18306371b5f0c6001b85\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2921, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "doc_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "ref_doc_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "chroma:document": "# Transformer XL\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, so we won't accept any new PRs changing its code. This model was deprecated due to security issues linked to `pickle.load`.\n\nWe recommend switching to more recent models for improved security.\n\nIn case you would still like to use `TransfoXL` in your experiments, we recommend using the [Hub checkpoint](https://huggingface.co/transfo-xl/transfo-xl-wt103) with a specific revision to ensure you are downloading safe files from the Hub.\n\nYou will need to set the environment variable `TRUST_REMOTE_CODE` to `True` in order to allow the\nusage of `pickle.load()`:\n\n```python\nimport os\nfrom transformers import TransfoXLTokenizer, TransfoXLLMHeadModel\n\nos.environ[\"TRUST_REMOTE_CODE\"] = \"True\"\n\ncheckpoint = 'transfo-xl/transfo-xl-wt103'\nrevision = '40a186da79458c9f9de846edfaea79c412137f97'\n\ntokenizer = TransfoXLTokenizer.from_pretrained(checkpoint, revision=revision)\nmodel = TransfoXLLMHeadModel.from_pretrained(checkpoint, revision=revision)\n```\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.35.0.\nYou can do so by running the following command: `pip install -U transformers==4.35.0`.\n\n</Tip>\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=transfo-xl\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-transfo--xl-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/transfo-xl-wt103\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Transformer-XL model was proposed in [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan\nSalakhutdinov. It's a causal (uni-directional) transformer with relative positioning (sinuso\u00efdal) embeddings which can\nreuse previously computed hidden-states to attend to longer context (memory). This model also uses adaptive softmax\ninputs and outputs (tied).\n\nThe abstract from the paper is the following:\n\n*Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the\nsetting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency\nbeyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a\nnovel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the\ncontext fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450%\nlonger than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+\ntimes faster than vanilla Transformers during evaluation."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f8fbb773-0f03-4b16-aea0-2cf4a7901c8a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f8fbb773-0f03-4b16-aea0-2cf4a7901c8a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl/\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0cf4b87-3f26-4ce2-af82-7f74ab96e191\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl/\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a5c23b428571e98b67a1340548c16bc1e47739fe122bf0a4eaedca03e3a8146e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b035a8be-30c7-46cc-972e-a342e23c7d39\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl/\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4c419e35089845185ce5dc91c944699eb9fb7bc6ab7d5056836a7ab38b3ad234\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"979b9866-ce56-4298-9e26-31c937319d7c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c7c7442136b16edd0ce74dbb3c2745722cc5cfe04297b008c536f202bf9950e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1868, \"end_char_idx\": 4701, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "doc_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "ref_doc_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "chroma:document": "It's a causal (uni-directional) transformer with relative positioning (sinuso\u00efdal) embeddings which can\nreuse previously computed hidden-states to attend to longer context (memory). This model also uses adaptive softmax\ninputs and outputs (tied).\n\nThe abstract from the paper is the following:\n\n*Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the\nsetting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency\nbeyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a\nnovel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the\ncontext fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450%\nlonger than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+\ntimes faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of\nbpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn\nTreebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably\ncoherent, novel text articles with thousands of tokens.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/kimiyoung/transformer-xl).\n\n## Usage tips\n\n- Transformer-XL uses relative sinusoidal positional embeddings. Padding can be done on the left or on the right. The\n  original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left.\n- Transformer-XL is one of the few models that has no sequence length limit.\n- Same as a regular GPT model, but introduces a recurrence mechanism for two consecutive segments (similar to a regular RNNs with two consecutive inputs). In this context, a segment is a number of consecutive tokens (for instance 512) that may span across multiple documents, and segments are fed in order to the model.\n- Basically, the hidden states of the previous segment are concatenated to the current input to compute the attention scores. This allows the model to pay attention to information that was in the previous segment as well as the current one. By stacking multiple attention layers, the receptive field can be increased to multiple previous segments.\n- This changes the positional embeddings to positional relative embeddings (as the regular positional embeddings would give the same results in the current input and the current hidden state at a given position) and needs to make some adjustments in the way attention scores are computed."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 979b9866-ce56-4298-9e26-31c937319d7c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"979b9866-ce56-4298-9e26-31c937319d7c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl/\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0cf4b87-3f26-4ce2-af82-7f74ab96e191\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl/\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a5c23b428571e98b67a1340548c16bc1e47739fe122bf0a4eaedca03e3a8146e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f8fbb773-0f03-4b16-aea0-2cf4a7901c8a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/transfo-xl/\", \"title\": \"Transformer XL\", \"tokens\": 1679, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4bf8e9664e7afdaa29edea3382a4f49f960e5ded173ca64b5c5e7c9de0e97e90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2922, \"end_char_idx\": 6171, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "doc_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "ref_doc_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "chroma:document": "Notably, we improve the state-of-the-art results of\nbpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn\nTreebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably\ncoherent, novel text articles with thousands of tokens.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/kimiyoung/transformer-xl).\n\n## Usage tips\n\n- Transformer-XL uses relative sinusoidal positional embeddings. Padding can be done on the left or on the right. The\n  original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left.\n- Transformer-XL is one of the few models that has no sequence length limit.\n- Same as a regular GPT model, but introduces a recurrence mechanism for two consecutive segments (similar to a regular RNNs with two consecutive inputs). In this context, a segment is a number of consecutive tokens (for instance 512) that may span across multiple documents, and segments are fed in order to the model.\n- Basically, the hidden states of the previous segment are concatenated to the current input to compute the attention scores. This allows the model to pay attention to information that was in the previous segment as well as the current one. By stacking multiple attention layers, the receptive field can be increased to multiple previous segments.\n- This changes the positional embeddings to positional relative embeddings (as the regular positional embeddings would give the same results in the current input and the current hidden state at a given position) and needs to make some adjustments in the way attention scores are computed.\n\n\n<Tip warning={true}>\n\nTransformerXL does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035)\n\n</Tip>\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## TransfoXLConfig\n\n[[autodoc]] TransfoXLConfig\n\n## TransfoXLTokenizer\n\n[[autodoc]] TransfoXLTokenizer\n    - save_vocabulary\n\n## TransfoXL specific outputs\n\n[[autodoc]] models.deprecated.transfo_xl.modeling_transfo_xl.TransfoXLModelOutput\n\n[[autodoc]] models.deprecated.transfo_xl.modeling_transfo_xl.TransfoXLLMHeadModelOutput\n\n[[autodoc]] models.deprecated.transfo_xl.modeling_tf_transfo_xl.TFTransfoXLModelOutput\n\n[[autodoc]] models.deprecated.transfo_xl.modeling_tf_transfo_xl.TFTransfoXLLMHeadModelOutput\n\n<frameworkcontent>\n<pt>\n\n## TransfoXLModel\n\n[[autodoc]] TransfoXLModel\n    - forward\n\n## TransfoXLLMHeadModel\n\n[[autodoc]] TransfoXLLMHeadModel\n    - forward\n\n## TransfoXLForSequenceClassification\n\n[[autodoc]] TransfoXLForSequenceClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFTransfoXLModel\n\n[[autodoc]] TFTransfoXLModel\n    - call\n\n## TFTransfoXLLMHeadModel\n\n[[autodoc]] TFTransfoXLLMHeadModel\n    - call\n\n## TFTransfoXLForSequenceClassification\n\n[[autodoc]] TFTransfoXLForSequenceClassification\n    - call\n\n</tf>\n</frameworkcontent>\n\n## Internal Layers\n\n[[autodoc]] AdaptiveEmbedding\n\n[[autodoc]] TFAdaptiveEmbedding"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c2a5eaf6-e041-459d-9f7c-772d2b140a10
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deta/", "title": "DETA", "tokens": 1005, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c2a5eaf6-e041-459d-9f7c-772d2b140a10\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta/\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"abd5b4a8-c91a-440e-9fd0-3019fd2b5821\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta/\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8da76146238a91143323bfab3c2d96a774f6ce96fe423a32ccb7db6c38ab7f37\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"175c7631-c1c2-40f2-9cbe-effd1abe1b62\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"942e832408de5b9f8a5d478a15063d8e2fa0c770fa1399e90031e39bcf7a1b34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3403, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "abd5b4a8-c91a-440e-9fd0-3019fd2b5821", "doc_id": "abd5b4a8-c91a-440e-9fd0-3019fd2b5821", "ref_doc_id": "abd5b4a8-c91a-440e-9fd0-3019fd2b5821", "chroma:document": "# DETA\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe DETA model was proposed in [NMS Strikes Back](https://arxiv.org/abs/2212.06137) by Jeffrey Ouyang-Zhang, Jang Hyun Cho, Xingyi Zhou, Philipp Kr\u00e4henb\u00fchl.\nDETA (short for Detection Transformers with Assignment) improves [Deformable DETR](deformable_detr) by replacing the one-to-one bipartite Hungarian matching loss\nwith one-to-many label assignments used in traditional detectors with non-maximum suppression (NMS). This leads to significant gains of up to 2.5 mAP.\n\nThe abstract from the paper is the following:\n\n*Detection Transformer (DETR) directly transforms queries to unique objects by using one-to-one bipartite matching during training and enables end-to-end object detection. Recently, these models have surpassed traditional detectors on COCO with undeniable elegance. However, they differ from traditional detectors in multiple designs, including model architecture and training schedules, and thus the effectiveness of one-to-one matching is not fully understood. In this work, we conduct a strict comparison between the one-to-one Hungarian matching in DETRs and the one-to-many label assignments in traditional detectors with non-maximum supervision (NMS). Surprisingly, we observe one-to-many assignments with NMS consistently outperform standard one-to-one matching under the same setting, with a significant gain of up to 2.5 mAP. Our detector that trains Deformable-DETR with traditional IoU-based label assignment achieved 50.2 COCO mAP within 12 epochs (1x schedule) with ResNet50 backbone, outperforming all existing traditional or transformer-based detectors in this setting. On multiple datasets, schedules, and architectures, we consistently show bipartite matching is unnecessary for performant detection transformers. Furthermore, we attribute the success of detection transformers to their expressive transformer architecture.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/deta_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> DETA overview. Taken from the <a href=\"https://arxiv.org/abs/2212.06137\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/jozhang97/DETA).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DETA.\n\n- Demo notebooks for DETA can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETA).\n- Scripts for finetuning [`DetaForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 175c7631-c1c2-40f2-9cbe-effd1abe1b62
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deta/", "title": "DETA", "tokens": 1005, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"175c7631-c1c2-40f2-9cbe-effd1abe1b62\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta/\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"abd5b4a8-c91a-440e-9fd0-3019fd2b5821\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta/\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8da76146238a91143323bfab3c2d96a774f6ce96fe423a32ccb7db6c38ab7f37\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c2a5eaf6-e041-459d-9f7c-772d2b140a10\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deta/\", \"title\": \"DETA\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"11261e1bae4eb721df448a522e38c4cfb417926f09da1e619b58090b37ac3d34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1696, \"end_char_idx\": 3678, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "abd5b4a8-c91a-440e-9fd0-3019fd2b5821", "doc_id": "abd5b4a8-c91a-440e-9fd0-3019fd2b5821", "ref_doc_id": "abd5b4a8-c91a-440e-9fd0-3019fd2b5821", "chroma:document": "Our detector that trains Deformable-DETR with traditional IoU-based label assignment achieved 50.2 COCO mAP within 12 epochs (1x schedule) with ResNet50 backbone, outperforming all existing traditional or transformer-based detectors in this setting. On multiple datasets, schedules, and architectures, we consistently show bipartite matching is unnecessary for performant detection transformers. Furthermore, we attribute the success of detection transformers to their expressive transformer architecture.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/deta_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> DETA overview. Taken from the <a href=\"https://arxiv.org/abs/2212.06137\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/jozhang97/DETA).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DETA.\n\n- Demo notebooks for DETA can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETA).\n- Scripts for finetuning [`DetaForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DetaConfig\n\n[[autodoc]] DetaConfig\n\n## DetaImageProcessor\n\n[[autodoc]] DetaImageProcessor\n    - preprocess\n    - post_process_object_detection\n\n## DetaModel\n\n[[autodoc]] DetaModel\n    - forward\n\n## DetaForObjectDetection\n\n[[autodoc]] DetaForObjectDetection\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0586bc02-81d7-4607-9d16-6d3dfad06b99
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/starcoder2/", "title": "Starcoder2", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0586bc02-81d7-4607-9d16-6d3dfad06b99\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2/\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"adde3766-db47-47cd-80b3-8f5a902ac4c1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2/\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0bb1b753da9eed875f21f2085c22caa72082d5648cabad7f6d548e82219c987c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"252e1bf1-b470-4a02-9747-145862fbe2f6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a64dc0b39a2a57ffb34c1d76c9296ba482aa2d91a2cabfa9c201ff13861ee2c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2653, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "adde3766-db47-47cd-80b3-8f5a902ac4c1", "doc_id": "adde3766-db47-47cd-80b3-8f5a902ac4c1", "ref_doc_id": "adde3766-db47-47cd-80b3-8f5a902ac4c1", "chroma:document": "# Starcoder2\n\n## Overview\n\nStarCoder2 is a family of open LLMs for code and comes in 3 different sizes with 3B, 7B and 15B parameters. The flagship StarCoder2-15B model is trained on over 4 trillion tokens and 600+ programming languages from The Stack v2. All models use Grouped Query Attention, a context window of 16,384 tokens with a sliding window attention of 4,096 tokens, and were trained using the Fill-in-the-Middle objective. The models have been released with the paper [StarCoder 2 and The Stack v2: The Next Generation](https://arxiv.org/abs/2402.19173) by Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krau\u00df, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Mu\u00f1oz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.\n\nThe abstract of the paper is the following:\n\n> The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 252e1bf1-b470-4a02-9747-145862fbe2f6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/starcoder2/", "title": "Starcoder2", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"252e1bf1-b470-4a02-9747-145862fbe2f6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2/\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"adde3766-db47-47cd-80b3-8f5a902ac4c1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2/\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0bb1b753da9eed875f21f2085c22caa72082d5648cabad7f6d548e82219c987c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0586bc02-81d7-4607-9d16-6d3dfad06b99\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/starcoder2/\", \"title\": \"Starcoder2\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"25be629165bf984a9789512a575b2e95dd86c06801c524f325ece94acd749ada\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1547, \"end_char_idx\": 4578, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "adde3766-db47-47cd-80b3-8f5a902ac4c1", "doc_id": "adde3766-db47-47cd-80b3-8f5a902ac4c1", "ref_doc_id": "adde3766-db47-47cd-80b3-8f5a902ac4c1", "chroma:document": "The abstract of the paper is the following:\n\n> The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size. Although DeepSeekCoder- 33B is the best-performing model at code completion for high-resource languages, we find that StarCoder2-15B outperforms it on math and code reasoning benchmarks, as well as several low-resource languages. We make the model weights available under an OpenRAIL license and ensure full transparency regarding the training data by releasing the SoftWare Heritage persistent IDentifiers (SWHIDs) of the source code data.\n## License\n\nThe models are licensed under the [BigCode OpenRAIL-M v1 license agreement](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement).\n \n## Usage tips\n\nThe StarCoder2 models can be found in the [HuggingFace hub](https://huggingface.co/collections/bigcode/starcoder2-65de6da6e87db3383572be1a). You can find some examples for inference and fine-tuning in StarCoder2's [GitHub repo](https://github.com/bigcode-project/starcoder2).\n\nThese ready-to-use checkpoints can be downloaded and used via the HuggingFace Hub:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"bigcode/starcoder2-7b\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"bigcode/starcoder2-7b\")\n\n>>> prompt = \"def print_hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=10, do_sample=False)\n>>> tokenizer.batch_decode(generated_ids)[0]\n'def print_hello_world():\\n    print(\"Hello World!\")\\n\\ndef print'\n```\n\n## Starcoder2Config\n\n[[autodoc]] Starcoder2Config\n\n## Starcoder2Model\n\n[[autodoc]] Starcoder2Model\n    - forward\n\n## Starcoder2ForCausalLM\n\n[[autodoc]] Starcoder2ForCausalLM\n    - forward\n\n## Starcoder2ForSequenceClassification\n\n[[autodoc]] Starcoder2ForSequenceClassification\n    - forward\n\n## Starcoder2ForTokenClassification\n\n[[autodoc]] Starcoder2ForTokenClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3d3578a7-b019-4be8-a97d-5002ada99a5d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3d3578a7-b019-4be8-a97d-5002ada99a5d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9b066302-c0a9-4902-a10b-fe696994d7fc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8a3e994d5e87979f2434c894341a3ac221a205229781257cc862731de6150126\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0f45d539-275f-4872-9731-28fc33993506\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ee65d2f4c8537c448c77e9c2366a62cb1168d338e2ed0bb0d815c8f26d30eb0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3307, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "doc_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "ref_doc_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "chroma:document": "# ELECTRA\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=electra\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-electra-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/electra_large_discriminator_squad2_512\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe ELECTRA model was proposed in the paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than\nGenerators](https://openreview.net/pdf?id=r1xMH1BtvB). ELECTRA is a new pretraining approach which trains two\ntransformer models: the generator and the discriminator. The generator's role is to replace tokens in a sequence, and\nis therefore trained as a masked language model. The discriminator, which is the model we're interested in, tries to\nidentify which tokens were replaced by the generator in the sequence.\n\nThe abstract from the paper is the following:\n\n*Masked language modeling (MLM) pretraining methods such as BERT corrupt the input by replacing some tokens with [MASK]\nand then train a model to reconstruct the original tokens. While they produce good results when transferred to\ndownstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a\nmore sample-efficient pretraining task called replaced token detection. Instead of masking the input, our approach\ncorrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead\nof training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that\npredicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments\ndemonstrate this new pretraining task is more efficient than MLM because the task is defined over all input tokens\nrather than just the small subset that was masked out. As a result, the contextual representations learned by our\napproach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are\nparticularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained\nusing 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale,\nwhere it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when\nusing the same amount of compute.*\n\nThis model was contributed by [lysandre](https://huggingface.co/lysandre). The original code can be found [here](https://github.com/google-research/electra).\n\n## Usage tips\n\n- ELECTRA is the pretraining approach, therefore there is nearly no changes done to the underlying model: BERT. The\n  only change is the separation of the embedding size and the hidden size: the embedding size is generally smaller,\n  while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from their\n  embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no projection\n  layer is used.\n- ELECTRA is a transformer model pretrained with the use of another (small) masked language model."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0f45d539-275f-4872-9731-28fc33993506
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0f45d539-275f-4872-9731-28fc33993506\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9b066302-c0a9-4902-a10b-fe696994d7fc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8a3e994d5e87979f2434c894341a3ac221a205229781257cc862731de6150126\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3d3578a7-b019-4be8-a97d-5002ada99a5d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e8e05a8d1c792542359200934293b905b154f41fa1529f30d880357b1b453be6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6a034c1c-1e0b-4b3b-8f3f-d443013e8633\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d56f0afe7b1d566508da7dc2f7bd94833395dcd218b5aaeadc147c935a3e9060\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "doc_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "ref_doc_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "chroma:document": "Then, instead\nof training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that\npredicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments\ndemonstrate this new pretraining task is more efficient than MLM because the task is defined over all input tokens\nrather than just the small subset that was masked out. As a result, the contextual representations learned by our\napproach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are\nparticularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained\nusing 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale,\nwhere it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when\nusing the same amount of compute.*\n\nThis model was contributed by [lysandre](https://huggingface.co/lysandre). The original code can be found [here](https://github.com/google-research/electra).\n\n## Usage tips\n\n- ELECTRA is the pretraining approach, therefore there is nearly no changes done to the underlying model: BERT. The\n  only change is the separation of the embedding size and the hidden size: the embedding size is generally smaller,\n  while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from their\n  embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no projection\n  layer is used.\n- ELECTRA is a transformer model pretrained with the use of another (small) masked language model. The inputs are corrupted by that language model, which takes an input text that is randomly masked and outputs a text in which ELECTRA has to predict which token is an original and which one has been replaced. Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps.\n- The ELECTRA checkpoints saved using [Google Research's implementation](https://github.com/google-research/electra)\n  contain both the generator and discriminator. The conversion script requires the user to name which model to export\n  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all\n  available ELECTRA models, however. This means that the discriminator may be loaded in the\n  [`ElectraForMaskedLM`] model, and the generator may be loaded in the\n  [`ElectraForPreTraining`] model (the classification head will be randomly initialized as it\n  doesn't exist in the generator).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## ElectraConfig\n\n[[autodoc]] ElectraConfig\n\n## ElectraTokenizer\n\n[[autodoc]] ElectraTokenizer\n\n## ElectraTokenizerFast\n\n[[autodoc]] ElectraTokenizerFast\n\n## Electra specific outputs\n\n[[autodoc]] models.electra.modeling_electra."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6a034c1c-1e0b-4b3b-8f3f-d443013e8633
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6a034c1c-1e0b-4b3b-8f3f-d443013e8633\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9b066302-c0a9-4902-a10b-fe696994d7fc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8a3e994d5e87979f2434c894341a3ac221a205229781257cc862731de6150126\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0f45d539-275f-4872-9731-28fc33993506\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d2f6e3a303c26607ab87798c2b358820a680fd035a7a75c03762b4994cac233c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d534999f-ee3e-4a1b-b132-14ebd3b6d39f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cf452d2274695614f192213ebfad351cbdf3a420e255c6d895d3e0e165190b1f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "doc_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "ref_doc_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "chroma:document": "The inputs are corrupted by that language model, which takes an input text that is randomly masked and outputs a text in which ELECTRA has to predict which token is an original and which one has been replaced. Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps.\n- The ELECTRA checkpoints saved using [Google Research's implementation](https://github.com/google-research/electra)\n  contain both the generator and discriminator. The conversion script requires the user to name which model to export\n  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all\n  available ELECTRA models, however. This means that the discriminator may be loaded in the\n  [`ElectraForMaskedLM`] model, and the generator may be loaded in the\n  [`ElectraForPreTraining`] model (the classification head will be randomly initialized as it\n  doesn't exist in the generator).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## ElectraConfig\n\n[[autodoc]] ElectraConfig\n\n## ElectraTokenizer\n\n[[autodoc]] ElectraTokenizer\n\n## ElectraTokenizerFast\n\n[[autodoc]] ElectraTokenizerFast\n\n## Electra specific outputs\n\n[[autodoc]] models.electra.modeling_electra.ElectraForPreTrainingOutput\n\n[[autodoc]] models.electra.modeling_tf_electra."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d534999f-ee3e-4a1b-b132-14ebd3b6d39f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d534999f-ee3e-4a1b-b132-14ebd3b6d39f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9b066302-c0a9-4902-a10b-fe696994d7fc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8a3e994d5e87979f2434c894341a3ac221a205229781257cc862731de6150126\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6a034c1c-1e0b-4b3b-8f3f-d443013e8633\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/electra/\", \"title\": \"ELECTRA\", \"tokens\": 1919, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"38766b8def14cb05264103e83f243c909706c33a7a6e29b4a307af9e9886515d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "doc_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "ref_doc_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "chroma:document": "Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps.\n- The ELECTRA checkpoints saved using [Google Research's implementation](https://github.com/google-research/electra)\n  contain both the generator and discriminator. The conversion script requires the user to name which model to export\n  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all\n  available ELECTRA models, however. This means that the discriminator may be loaded in the\n  [`ElectraForMaskedLM`] model, and the generator may be loaded in the\n  [`ElectraForPreTraining`] model (the classification head will be randomly initialized as it\n  doesn't exist in the generator).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## ElectraConfig\n\n[[autodoc]] ElectraConfig\n\n## ElectraTokenizer\n\n[[autodoc]] ElectraTokenizer\n\n## ElectraTokenizerFast\n\n[[autodoc]] ElectraTokenizerFast\n\n## Electra specific outputs\n\n[[autodoc]] models.electra.modeling_electra.ElectraForPreTrainingOutput\n\n[[autodoc]] models.electra.modeling_tf_electra.TFElectraForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## ElectraModel\n\n[[autodoc]] ElectraModel\n    - forward\n\n## ElectraForPreTraining\n\n[[autodoc]] ElectraForPreTraining\n    - forward\n\n## ElectraForCausalLM\n\n[[autodoc]] ElectraForCausalLM\n    - forward\n\n## ElectraForMaskedLM\n\n[[autodoc]] ElectraForMaskedLM\n    - forward\n\n## ElectraForSequenceClassification\n\n[[autodoc]] ElectraForSequenceClassification\n    - forward\n\n## ElectraForMultipleChoice\n\n[[autodoc]] ElectraForMultipleChoice\n    - forward\n\n## ElectraForTokenClassification\n\n[[autodoc]] ElectraForTokenClassification\n    - forward\n\n## ElectraForQuestionAnswering\n\n[[autodoc]] ElectraForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFElectraModel\n\n[[autodoc]] TFElectraModel\n    - call\n\n## TFElectraForPreTraining\n\n[[autodoc]] TFElectraForPreTraining\n    - call\n\n## TFElectraForMaskedLM\n\n[[autodoc]] TFElectraForMaskedLM\n    - call\n\n## TFElectraForSequenceClassification\n\n[[autodoc]] TFElectraForSequenceClassification\n    - call\n\n## TFElectraForMultipleChoice\n\n[[autodoc]] TFElectraForMultipleChoice\n    - call\n\n## TFElectraForTokenClassification\n\n[[autodoc]] TFElectraForTokenClassification\n    - call\n\n## TFElectraForQuestionAnswering\n\n[[autodoc]] TFElectraForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxElectraModel\n\n[[autodoc]] FlaxElectraModel\n    - __call__\n\n## FlaxElectraForPreTraining\n\n[[autodoc]] FlaxElectraForPreTraining\n    - __call__\n\n## FlaxElectraForCausalLM\n\n[[autodoc]] FlaxElectraForCausalLM\n    - __call__\n\n## FlaxElectraForMaskedLM\n\n[[autodoc]] FlaxElectraForMaskedLM\n    - __call__\n\n## FlaxElectraForSequenceClassification\n\n[[autodoc]] FlaxElectraForSequenceClassification\n    - __call__\n\n## FlaxElectraForMultipleChoice\n\n[[autodoc]] FlaxElectraForMultipleChoice\n    - __call__\n\n## FlaxElectraForTokenClassification\n\n[[autodoc]] FlaxElectraForTokenClassification\n    - __call__\n\n## FlaxElectraForQuestionAnswering\n\n[[autodoc]] FlaxElectraForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c957a68c-8f3f-4d74-a678-cde38acc33dd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/", "title": "RoBERTa-PreLayerNorm", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c957a68c-8f3f-4d74-a678-cde38acc33dd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3cd683a7-efa7-4825-b001-0474c1edc100\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dc6b78d56dbe8cd66a97b535e65027e9c2ad324ad709f5de5d7209740a9d7cce\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1e2a2cf7-a9f7-4e73-ba08-f30a77bba942\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"34ec90fcec75459229a09d392332aa98ceab3b1400c7f1bf6268f54baa0d3797\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3cd683a7-efa7-4825-b001-0474c1edc100", "doc_id": "3cd683a7-efa7-4825-b001-0474c1edc100", "ref_doc_id": "3cd683a7-efa7-4825-b001-0474c1edc100", "chroma:document": "# RoBERTa-PreLayerNorm\n\n## Overview\n\nThe RoBERTa-PreLayerNorm model was proposed in [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://arxiv.org/abs/1904.01038) by Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli.\nIt is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).\n\nThe abstract from the paper is the following:\n\n*fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs.*\n\nThis model was contributed by [andreasmaden](https://huggingface.co/andreasmadsen).\nThe original code can be found [here](https://github.com/princeton-nlp/DinkyTrain).\n\n## Usage tips\n\n- The implementation is the same as [Roberta](roberta) except instead of using _Add and Norm_ it does _Norm and Add_. _Add_ and _Norm_ refers to the Addition and LayerNormalization as described in [Attention Is All You Need](https://arxiv.org/abs/1706.03762).\n- This is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1e2a2cf7-a9f7-4e73-ba08-f30a77bba942
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/", "title": "RoBERTa-PreLayerNorm", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1e2a2cf7-a9f7-4e73-ba08-f30a77bba942\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3cd683a7-efa7-4825-b001-0474c1edc100\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dc6b78d56dbe8cd66a97b535e65027e9c2ad324ad709f5de5d7209740a9d7cce\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c957a68c-8f3f-4d74-a678-cde38acc33dd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/\", \"title\": \"RoBERTa-PreLayerNorm\", \"tokens\": 1289, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7fb50404493f04d8e56ee97b2fa08587d20ab8ba10b14ca4b014b822a464354c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3cd683a7-efa7-4825-b001-0474c1edc100", "doc_id": "3cd683a7-efa7-4825-b001-0474c1edc100", "ref_doc_id": "3cd683a7-efa7-4825-b001-0474c1edc100", "chroma:document": "It is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).\n\nThe abstract from the paper is the following:\n\n*fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs.*\n\nThis model was contributed by [andreasmaden](https://huggingface.co/andreasmadsen).\nThe original code can be found [here](https://github.com/princeton-nlp/DinkyTrain).\n\n## Usage tips\n\n- The implementation is the same as [Roberta](roberta) except instead of using _Add and Norm_ it does _Norm and Add_. _Add_ and _Norm_ refers to the Addition and LayerNormalization as described in [Attention Is All You Need](https://arxiv.org/abs/1706.03762).\n- This is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Causal language modeling task guide](./tasks/language_modeling)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## RobertaPreLayerNormConfig\n\n[[autodoc]] RobertaPreLayerNormConfig\n\n<frameworkcontent>\n<pt>\n\n## RobertaPreLayerNormModel\n\n[[autodoc]] RobertaPreLayerNormModel\n    - forward\n\n## RobertaPreLayerNormForCausalLM\n\n[[autodoc]] RobertaPreLayerNormForCausalLM\n    - forward\n\n## RobertaPreLayerNormForMaskedLM\n\n[[autodoc]] RobertaPreLayerNormForMaskedLM\n    - forward\n\n## RobertaPreLayerNormForSequenceClassification\n\n[[autodoc]] RobertaPreLayerNormForSequenceClassification\n    - forward\n\n## RobertaPreLayerNormForMultipleChoice\n\n[[autodoc]] RobertaPreLayerNormForMultipleChoice\n    - forward\n\n## RobertaPreLayerNormForTokenClassification\n\n[[autodoc]] RobertaPreLayerNormForTokenClassification\n    - forward\n\n## RobertaPreLayerNormForQuestionAnswering\n\n[[autodoc]] RobertaPreLayerNormForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFRobertaPreLayerNormModel\n\n[[autodoc]] TFRobertaPreLayerNormModel\n    - call\n\n## TFRobertaPreLayerNormForCausalLM\n\n[[autodoc]] TFRobertaPreLayerNormForCausalLM\n    - call\n\n## TFRobertaPreLayerNormForMaskedLM\n\n[[autodoc]] TFRobertaPreLayerNormForMaskedLM\n    - call\n\n## TFRobertaPreLayerNormForSequenceClassification\n\n[[autodoc]] TFRobertaPreLayerNormForSequenceClassification\n    - call\n\n## TFRobertaPreLayerNormForMultipleChoice\n\n[[autodoc]] TFRobertaPreLayerNormForMultipleChoice\n    - call\n\n## TFRobertaPreLayerNormForTokenClassification\n\n[[autodoc]] TFRobertaPreLayerNormForTokenClassification\n    - call\n\n## TFRobertaPreLayerNormForQuestionAnswering\n\n[[autodoc]] TFRobertaPreLayerNormForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxRobertaPreLayerNormModel\n\n[[autodoc]] FlaxRobertaPreLayerNormModel\n    - __call__\n\n## FlaxRobertaPreLayerNormForCausalLM\n\n[[autodoc]] FlaxRobertaPreLayerNormForCausalLM\n    - __call__\n\n## FlaxRobertaPreLayerNormForMaskedLM\n\n[[autodoc]] FlaxRobertaPreLayerNormForMaskedLM\n    - __call__\n\n## FlaxRobertaPreLayerNormForSequenceClassification\n\n[[autodoc]] FlaxRobertaPreLayerNormForSequenceClassification\n    - __call__\n\n## FlaxRobertaPreLayerNormForMultipleChoice\n\n[[autodoc]] FlaxRobertaPreLayerNormForMultipleChoice\n    - __call__\n\n## FlaxRobertaPreLayerNormForTokenClassification\n\n[[autodoc]] FlaxRobertaPreLayerNormForTokenClassification\n    - __call__\n\n## FlaxRobertaPreLayerNormForQuestionAnswering\n\n[[autodoc]] FlaxRobertaPreLayerNormForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b4bf5881-7e3e-4e4f-9b26-c9a8f8e58fcd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilevitv2/", "title": "MobileViTV2", "tokens": 862, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b4bf5881-7e3e-4e4f-9b26-c9a8f8e58fcd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevitv2/\", \"title\": \"MobileViTV2\", \"tokens\": 862, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5e4a2bd3-9cf4-42e0-aa6a-efadf3efdc9f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevitv2/\", \"title\": \"MobileViTV2\", \"tokens\": 862, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6c89f466767cbf94faa452a3be3239881df0c5571a01e3e81b2abd36f9014f77\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2960, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5e4a2bd3-9cf4-42e0-aa6a-efadf3efdc9f", "doc_id": "5e4a2bd3-9cf4-42e0-aa6a-efadf3efdc9f", "ref_doc_id": "5e4a2bd3-9cf4-42e0-aa6a-efadf3efdc9f", "chroma:document": "# MobileViTV2\n\n## Overview\n\nThe MobileViTV2 model was proposed in [Separable Self-attention for Mobile Vision Transformers](https://arxiv.org/abs/2206.02680) by Sachin Mehta and Mohammad Rastegari.\n\nMobileViTV2 is the second version of MobileViT, constructed by replacing the multi-headed self-attention in MobileViT with separable self-attention.\n\nThe abstract from the paper is the following:\n\n*Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network-based models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires O(k2) time complexity with respect to the number of tokens (or patches) k. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrained devices. This paper introduces a separable self-attention method with linear complexity, i.e. O(k). A simple yet effective characteristic of the proposed method is that it uses element-wise operations for computing self-attention, making it a good choice for resource-constrained devices. The improved model, MobileViTV2, is state-of-the-art on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTV2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileViT by about 1% while running 3.2\u00d7 faster on a mobile device.*\n\nThis model was contributed by [shehan97](https://huggingface.co/shehan97).\nThe original code can be found [here](https://github.com/apple/ml-cvnets).\n\n## Usage tips\n\n- MobileViTV2 is more like a CNN than a Transformer model. It does not work on sequence data but on batches of images. Unlike ViT, there are no embeddings. The backbone model outputs a feature map.\n- One can use [`MobileViTImageProcessor`] to prepare images for the model. Note that if you do your own preprocessing, the pretrained checkpoints expect images to be in BGR pixel order (not RGB).\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes).\n- The segmentation model uses a [DeepLabV3](https://arxiv.org/abs/1706.05587) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).\n\n## MobileViTV2Config\n\n[[autodoc]] MobileViTV2Config\n\n## MobileViTV2Model\n\n[[autodoc]] MobileViTV2Model\n    - forward\n\n## MobileViTV2ForImageClassification\n\n[[autodoc]] MobileViTV2ForImageClassification\n    - forward\n\n## MobileViTV2ForSemanticSegmentation\n\n[[autodoc]] MobileViTV2ForSemanticSegmentation\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f0bf9c21-a07f-44c3-a5af-c6224d1105e2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f0bf9c21-a07f-44c3-a5af-c6224d1105e2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf3d26cd-736b-4b34-b57a-9b8dc7a421f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b318c20228b6d10f08e9e72e3c5a22a6419bcaf0767cbfac2c4ab33a9bb50fc1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"41aee283-7d33-4a2c-a3a7-b08749cfa2ca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"188b9d991439a2fed573f6a51c522ea5cc660eef77e7d94250fb2192ff3b67c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2725, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "doc_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "ref_doc_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "chroma:document": "# Grounding DINO\n\n## Overview\n\nThe Grounding DINO model was proposed in [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://arxiv.org/abs/2303.05499) by Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang. Grounding DINO extends a closed-set object detection model with a text encoder, enabling open-set object detection. The model achieves remarkable results, such as 52.5 AP on COCO zero-shot.\n\nThe abstract from the paper is the following:\n\n*In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a 52.5 AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean 26.1 AP.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grouding_dino_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Grounding DINO overview. Taken from the <a href=\"https://arxiv.org/abs/2303.05499\">original paper</a>. </small>\n\nThis model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/IDEA-Research/GroundingDINO).\n\n## Usage tips\n\n- One can use [`GroundingDinoProcessor`] to prepare image-text pairs for the model.\n- To separate classes in the text use a period e.g. \"a cat. a dog.\"\n- When using multiple classes (e.g. `\"a cat. a dog.\"`), use `post_process_grounded_object_detection` from [`GroundingDinoProcessor`] to post process outputs. Since, the labels returned from `post_process_object_detection` represent the indices from the model dimension where prob > threshold."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 41aee283-7d33-4a2c-a3a7-b08749cfa2ca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"41aee283-7d33-4a2c-a3a7-b08749cfa2ca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf3d26cd-736b-4b34-b57a-9b8dc7a421f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b318c20228b6d10f08e9e72e3c5a22a6419bcaf0767cbfac2c4ab33a9bb50fc1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0bf9c21-a07f-44c3-a5af-c6224d1105e2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e44aea0b6b0d14ee59e07350a19a6471ed4ff40f1cf64dc5691bffa177669e61\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d002e1bb-1807-4eb8-a0ab-dc65e2c02634\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fef42d10ad6741160fdb6ec00f22268db4231f8acca715345a7600180a4b044d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1225, \"end_char_idx\": 3373, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "doc_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "ref_doc_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "chroma:document": "While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a 52.5 AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean 26.1 AP.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grouding_dino_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Grounding DINO overview. Taken from the <a href=\"https://arxiv.org/abs/2303.05499\">original paper</a>. </small>\n\nThis model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/IDEA-Research/GroundingDINO).\n\n## Usage tips\n\n- One can use [`GroundingDinoProcessor`] to prepare image-text pairs for the model.\n- To separate classes in the text use a period e.g. \"a cat. a dog.\"\n- When using multiple classes (e.g. `\"a cat. a dog.\"`), use `post_process_grounded_object_detection` from [`GroundingDinoProcessor`] to post process outputs. Since, the labels returned from `post_process_object_detection` represent the indices from the model dimension where prob > threshold.\n\nHere's how to use the model for zero-shot object detection:\n\n```python\n>>> import requests\n\n>>> import torch\n>>> from PIL import Image\n>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection\n\n>>> model_id = \"IDEA-Research/grounding-dino-tiny\"\n>>> device = \"cuda\"\n\n>>> processor = AutoProcessor.from_pretrained(model_id)\n>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id).to(device)\n\n>>> image_url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(image_url, stream=True).raw)\n>>> # Check for cats and remote controls\n>>> text = \"a cat. a remote control.\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d002e1bb-1807-4eb8-a0ab-dc65e2c02634
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d002e1bb-1807-4eb8-a0ab-dc65e2c02634\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf3d26cd-736b-4b34-b57a-9b8dc7a421f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b318c20228b6d10f08e9e72e3c5a22a6419bcaf0767cbfac2c4ab33a9bb50fc1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"41aee283-7d33-4a2c-a3a7-b08749cfa2ca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"267bff8f93e9552fb34b955b40655edd4af50f24ae6742278f322cc4e6d1e4c7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f5d287b7-8f19-4c3d-ab05-07c10e1ff026\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"38840528246ff7cd0f6775955bb289d6e21230694760df90abdad62daa72bd63\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1959, \"end_char_idx\": 4651, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "doc_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "ref_doc_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2303.05499\">original paper</a>. </small>\n\nThis model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/IDEA-Research/GroundingDINO).\n\n## Usage tips\n\n- One can use [`GroundingDinoProcessor`] to prepare image-text pairs for the model.\n- To separate classes in the text use a period e.g. \"a cat. a dog.\"\n- When using multiple classes (e.g. `\"a cat. a dog.\"`), use `post_process_grounded_object_detection` from [`GroundingDinoProcessor`] to post process outputs. Since, the labels returned from `post_process_object_detection` represent the indices from the model dimension where prob > threshold.\n\nHere's how to use the model for zero-shot object detection:\n\n```python\n>>> import requests\n\n>>> import torch\n>>> from PIL import Image\n>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection\n\n>>> model_id = \"IDEA-Research/grounding-dino-tiny\"\n>>> device = \"cuda\"\n\n>>> processor = AutoProcessor.from_pretrained(model_id)\n>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id).to(device)\n\n>>> image_url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(image_url, stream=True).raw)\n>>> # Check for cats and remote controls\n>>> text = \"a cat. a remote control.\"\n\n>>> inputs = processor(images=image, text=text, return_tensors=\"pt\").to(device)\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> results = processor.post_process_grounded_object_detection(\n...     outputs,\n...     inputs.input_ids,\n...     box_threshold=0.4,\n...     text_threshold=0.3,\n...     target_sizes=[image.size[::-1]]\n... )\n>>> print(results)\n[{'boxes': tensor([[344.6959,  23.1090, 637.1833, 374.2751],\n        [ 12.2666,  51.9145, 316.8582, 472.4392],\n        [ 38.5742,  70.0015, 176.7838, 118.1806]], device='cuda:0'),\n  'labels': ['a cat', 'a cat', 'a remote control'],\n  'scores': tensor([0.4785, 0.4381, 0.4776], device='cuda:0')}]\n```\n\n## Grounded SAM\n\nOne can combine Grounding DINO with the [Segment Anything](sam) model for text-based mask generation as introduced in [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159). You can refer to this [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Grounding%20DINO/GroundingDINO_with_Segment_Anything.ipynb) \ud83c\udf0d for details.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grounded_sam.png\"\nalt=\"drawing\" width=\"900\"/>\n\n<small> Grounded SAM overview."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f5d287b7-8f19-4c3d-ab05-07c10e1ff026
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f5d287b7-8f19-4c3d-ab05-07c10e1ff026\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf3d26cd-736b-4b34-b57a-9b8dc7a421f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b318c20228b6d10f08e9e72e3c5a22a6419bcaf0767cbfac2c4ab33a9bb50fc1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d002e1bb-1807-4eb8-a0ab-dc65e2c02634\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/grounding-dino/\", \"title\": \"Grounding DINO\", \"tokens\": 1632, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"94806d86f4351f1e84a48fe524911e513215ddf55821f4ed38e0fc0f7ed6a1e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3355, \"end_char_idx\": 5788, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "doc_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "ref_doc_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "chroma:document": "a remote control.\"\n\n>>> inputs = processor(images=image, text=text, return_tensors=\"pt\").to(device)\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> results = processor.post_process_grounded_object_detection(\n...     outputs,\n...     inputs.input_ids,\n...     box_threshold=0.4,\n...     text_threshold=0.3,\n...     target_sizes=[image.size[::-1]]\n... )\n>>> print(results)\n[{'boxes': tensor([[344.6959,  23.1090, 637.1833, 374.2751],\n        [ 12.2666,  51.9145, 316.8582, 472.4392],\n        [ 38.5742,  70.0015, 176.7838, 118.1806]], device='cuda:0'),\n  'labels': ['a cat', 'a cat', 'a remote control'],\n  'scores': tensor([0.4785, 0.4381, 0.4776], device='cuda:0')}]\n```\n\n## Grounded SAM\n\nOne can combine Grounding DINO with the [Segment Anything](sam) model for text-based mask generation as introduced in [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159). You can refer to this [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Grounding%20DINO/GroundingDINO_with_Segment_Anything.ipynb) \ud83c\udf0d for details.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grounded_sam.png\"\nalt=\"drawing\" width=\"900\"/>\n\n<small> Grounded SAM overview. Taken from the <a href=\"https://github.com/IDEA-Research/Grounded-Segment-Anything\">original repository</a>. </small>\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Grounding DINO. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Demo notebooks regarding inference with Grounding DINO as well as combining it with [SAM](sam) can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Grounding%20DINO). \ud83c\udf0e\n\n## GroundingDinoImageProcessor\n\n[[autodoc]] GroundingDinoImageProcessor\n    - preprocess\n    - post_process_object_detection\n\n## GroundingDinoProcessor\n\n[[autodoc]] GroundingDinoProcessor\n    - post_process_grounded_object_detection\n\n## GroundingDinoConfig\n\n[[autodoc]] GroundingDinoConfig\n\n## GroundingDinoModel\n\n[[autodoc]] GroundingDinoModel\n    - forward\n\n## GroundingDinoForObjectDetection\n\n[[autodoc]] GroundingDinoForObjectDetection\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 59a21c2c-156e-44ef-b823-c33252813c55
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"59a21c2c-156e-44ef-b823-c33252813c55\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2567f576-ca06-4dee-8abb-fc80b346a8b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fae59e5fedda3ee339831bf7dda4f082d89e7b573230fc361cf09ae6b25f090a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d8da0307-f72b-42ed-bfdc-21b4eee253e2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"00549dccda6ac18f7182823f17a5f89a9216ec2cf4ea2261346c11eadd9215f7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3075, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "doc_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "ref_doc_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "chroma:document": "# MarianMT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=marian\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-marian-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/opus-mt-zh-en\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nA framework for translation models, using the same models as BART. Translations should be similar, but not identical to output in the test set linked to in each model card.\nThis model was contributed by [sshleifer](https://huggingface.co/sshleifer).\n\n\n## Implementation Notes\n\n- Each model is about 298 MB on disk, there are more than 1,000 models.\n- The list of supported language pairs can be found [here](https://huggingface.co/Helsinki-NLP).\n- Models were originally trained by [J\u00f6rg Tiedemann](https://researchportal.helsinki.fi/en/persons/j%C3%B6rg-tiedemann) using the [Marian](https://marian-nmt.github.io/) C++ library, which supports fast training and translation.\n- All models are transformer encoder-decoders with 6 layers in each component. Each model's performance is documented\n  in a model card.\n- The 80 opus models that require BPE preprocessing are not supported.\n- The modeling code is the same as [`BartForConditionalGeneration`] with a few minor modifications:\n\n  - static (sinusoid) positional embeddings (`MarianConfig.static_position_embeddings=True`)\n  - no layernorm_embedding (`MarianConfig.normalize_embedding=False`)\n  - the model starts generating with `pad_token_id` (which has 0 as a token_embedding) as the prefix (Bart uses\n    `<s/>`),\n- Code to bulk convert models can be found in `convert_marian_to_pytorch.py`.\n\n\n## Naming\n\n- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`\n- The language codes used to name models are inconsistent. Two digit codes can usually be found [here](https://developers.google.com/admin-sdk/directory/v1/languages), three digit codes require googling \"language\n  code {code}\".\n- Codes formatted like `es_AR` are usually `code_{region}`. That one is Spanish from Argentina.\n- The models were converted in two stages. The first 1000 models use ISO-639-2 codes to identify languages, the second\n  group use a combination of ISO-639-5 codes and ISO-639-2 codes.\n\n\n## Examples\n\n- Since Marian models are smaller than many other translation models available in the library, they can be useful for\n  fine-tuning experiments and integration tests.\n- [Fine-tune on GPU](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/train_distil_marian_enro.sh)\n\n## Multilingual Models\n\n- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`:\n- If a model can output multiple languages, and you should specify a language code by prepending the desired output\n  language to the `src_text`.\n- You can see a models's supported language codes in its model card, under target constituents, like in [opus-mt-en-roa](https://huggingface.co/Helsinki-NLP/opus-mt-en-roa)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d8da0307-f72b-42ed-bfdc-21b4eee253e2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d8da0307-f72b-42ed-bfdc-21b4eee253e2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2567f576-ca06-4dee-8abb-fc80b346a8b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fae59e5fedda3ee339831bf7dda4f082d89e7b573230fc361cf09ae6b25f090a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"59a21c2c-156e-44ef-b823-c33252813c55\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e400a98596cb7dc08da79759c944cd2c09fe4f1ac0e692823c3cc3db870de945\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e3414972-83a8-4973-b0a4-08c6cd9ecb33\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b763b721895cd79422aa777b710922e81e61a50d45b95a0986078267d9497f71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1739, \"end_char_idx\": 4290, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "doc_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "ref_doc_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "chroma:document": "## Naming\n\n- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`\n- The language codes used to name models are inconsistent. Two digit codes can usually be found [here](https://developers.google.com/admin-sdk/directory/v1/languages), three digit codes require googling \"language\n  code {code}\".\n- Codes formatted like `es_AR` are usually `code_{region}`. That one is Spanish from Argentina.\n- The models were converted in two stages. The first 1000 models use ISO-639-2 codes to identify languages, the second\n  group use a combination of ISO-639-5 codes and ISO-639-2 codes.\n\n\n## Examples\n\n- Since Marian models are smaller than many other translation models available in the library, they can be useful for\n  fine-tuning experiments and integration tests.\n- [Fine-tune on GPU](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/train_distil_marian_enro.sh)\n\n## Multilingual Models\n\n- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`:\n- If a model can output multiple languages, and you should specify a language code by prepending the desired output\n  language to the `src_text`.\n- You can see a models's supported language codes in its model card, under target constituents, like in [opus-mt-en-roa](https://huggingface.co/Helsinki-NLP/opus-mt-en-roa).\n- Note that if a model is only multilingual on the source side, like `Helsinki-NLP/opus-mt-roa-en`, no language\n  codes are required.\n\nNew multi-lingual models from the [Tatoeba-Challenge repo](https://github.com/Helsinki-NLP/Tatoeba-Challenge)\nrequire 3 character language codes:\n\n```python\n>>> from transformers import MarianMTModel, MarianTokenizer\n\n>>> src_text = [\n...     \">>fra<< this is a sentence in english that we want to translate to french\",\n...     \">>por<< This should go to portuguese\",\n...     \">>esp<< And this to Spanish\",\n... ]\n\n>>> model_name = \"Helsinki-NLP/opus-mt-en-roa\"\n>>> tokenizer = MarianTokenizer.from_pretrained(model_name)\n>>> print(tokenizer.supported_language_codes)\n['>>zlm_Latn<<', '>>mfe<<', '>>hat<<', '>>pap<<', '>>ast<<', '>>cat<<', '>>ind<<', '>>glg<<', '>>wln<<', '>>spa<<', '>>fra<<', '>>ron<<', '>>por<<', '>>ita<<', '>>oci<<', '>>arg<<', '>>min<<']\n\n>>> model = MarianMTModel.from_pretrained(model_name)\n>>> translated = model.generate(**tokenizer(src_text, return_tensors=\"pt\", padding=True))\n>>> [tokenizer.decode(t, skip_special_tokens=True) for t in translated]\n[\"c'est une phrase en anglais que nous voulons traduire en fran\u00e7ais\",\n 'Isto deve ir para o portugu\u00eas."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e3414972-83a8-4973-b0a4-08c6cd9ecb33
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e3414972-83a8-4973-b0a4-08c6cd9ecb33\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2567f576-ca06-4dee-8abb-fc80b346a8b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fae59e5fedda3ee339831bf7dda4f082d89e7b573230fc361cf09ae6b25f090a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d8da0307-f72b-42ed-bfdc-21b4eee253e2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"266b996c329a4c9e7af1e40d698f4d5d325b453fcf9942581115b95ba78e3f79\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"063bd867-5a5c-4ea6-b607-60aef33e2eb2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9e8e09492690d1c849851363e67b89752b7278f8fc4d307fe4fff7848cbf4a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2902, \"end_char_idx\": 6221, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "doc_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "ref_doc_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "chroma:document": "- You can see a models's supported language codes in its model card, under target constituents, like in [opus-mt-en-roa](https://huggingface.co/Helsinki-NLP/opus-mt-en-roa).\n- Note that if a model is only multilingual on the source side, like `Helsinki-NLP/opus-mt-roa-en`, no language\n  codes are required.\n\nNew multi-lingual models from the [Tatoeba-Challenge repo](https://github.com/Helsinki-NLP/Tatoeba-Challenge)\nrequire 3 character language codes:\n\n```python\n>>> from transformers import MarianMTModel, MarianTokenizer\n\n>>> src_text = [\n...     \">>fra<< this is a sentence in english that we want to translate to french\",\n...     \">>por<< This should go to portuguese\",\n...     \">>esp<< And this to Spanish\",\n... ]\n\n>>> model_name = \"Helsinki-NLP/opus-mt-en-roa\"\n>>> tokenizer = MarianTokenizer.from_pretrained(model_name)\n>>> print(tokenizer.supported_language_codes)\n['>>zlm_Latn<<', '>>mfe<<', '>>hat<<', '>>pap<<', '>>ast<<', '>>cat<<', '>>ind<<', '>>glg<<', '>>wln<<', '>>spa<<', '>>fra<<', '>>ron<<', '>>por<<', '>>ita<<', '>>oci<<', '>>arg<<', '>>min<<']\n\n>>> model = MarianMTModel.from_pretrained(model_name)\n>>> translated = model.generate(**tokenizer(src_text, return_tensors=\"pt\", padding=True))\n>>> [tokenizer.decode(t, skip_special_tokens=True) for t in translated]\n[\"c'est une phrase en anglais que nous voulons traduire en fran\u00e7ais\",\n 'Isto deve ir para o portugu\u00eas.',\n 'Y esto al espa\u00f1ol']\n```\n\nHere is the code to see all available pretrained models on the hub:\n\n```python\nfrom huggingface_hub import list_models\n\nmodel_list = list_models()\norg = \"Helsinki-NLP\"\nmodel_ids = [x.id for x in model_list if x.id.startswith(org)]\nsuffix = [x.split(\"/\")[1] for x in model_ids]\nold_style_multi_models = [f\"{org}/{s}\" for s in suffix if s != s.lower()]\n```\n\n## Old Style Multi-Lingual Models\n\nThese are the old style multi-lingual models ported from the OPUS-MT-Train repo: and the members of each language\ngroup:\n\n```python no-style\n['Helsinki-NLP/opus-mt-NORTH_EU-NORTH_EU',\n 'Helsinki-NLP/opus-mt-ROMANCE-en',\n 'Helsinki-NLP/opus-mt-SCANDINAVIA-SCANDINAVIA',\n 'Helsinki-NLP/opus-mt-de-ZH',\n 'Helsinki-NLP/opus-mt-en-CELTIC',\n 'Helsinki-NLP/opus-mt-en-ROMANCE',\n 'Helsinki-NLP/opus-mt-es-NORWAY',\n 'Helsinki-NLP/opus-mt-fi-NORWAY',\n 'Helsinki-NLP/opus-mt-fi-ZH',\n 'Helsinki-NLP/opus-mt-fi_nb_no_nn_ru_sv_en-SAMI',\n 'Helsinki-NLP/opus-mt-sv-NORWAY',\n 'Helsinki-NLP/opus-mt-sv-ZH']\nGROUP_MEMBERS = {\n 'ZH': ['cmn', 'cn', 'yue', 'ze_zh', 'zh_cn', 'zh_CN', 'zh_HK', 'zh_tw', 'zh_TW', 'zh_yue', 'zhs', 'zht', 'zh'],\n 'ROMANCE': ['fr', 'fr_BE', 'fr_CA', 'fr_FR', 'wa', 'frp', 'oc', 'ca', 'rm', 'lld', 'fur', 'lij', 'lmo', 'es', 'es_AR', 'es_CL', 'es_CO', 'es_CR', 'es_DO', 'es_EC', 'es_ES', 'es_GT', 'es_HN', 'es_MX', 'es_NI', 'es_PA', 'es_PE', 'es_PR', 'es_SV', 'es_UY', 'es_VE', 'pt', 'pt_br', 'pt_BR', 'pt_PT', 'gl', 'lad', 'an', 'mwl', 'it', 'it_IT', 'co', 'nap', 'scn', 'vec', 'sc', 'ro', 'la'],\n 'NORTH_EU': ['de', 'nl', 'fy', 'af', 'da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],\n 'SCANDINAVIA': ['da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],\n 'SAMI': ['se', 'sma', 'smj', 'smn', 'sms'],\n 'NORWAY': ['nb_NO', 'nb', 'nn_NO', 'nn', 'nog', 'no_nb', 'no'],\n 'CELTIC': ['ga', 'cy', 'br', 'gd', 'kw', 'gv']\n}\n```\n\nExample of translating english to many romance languages, using old-style 2 character language codes"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 063bd867-5a5c-4ea6-b607-60aef33e2eb2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"063bd867-5a5c-4ea6-b607-60aef33e2eb2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2567f576-ca06-4dee-8abb-fc80b346a8b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fae59e5fedda3ee339831bf7dda4f082d89e7b573230fc361cf09ae6b25f090a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e3414972-83a8-4973-b0a4-08c6cd9ecb33\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/marian/\", \"title\": \"MarianMT\", \"tokens\": 2443, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"299b371787cd2a53ecb797da527760a513aeb04bc2caae36567f9227667b2fb8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6224, \"end_char_idx\": 7781, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "doc_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "ref_doc_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "chroma:document": "```python\n>>> from transformers import MarianMTModel, MarianTokenizer\n\n>>> src_text = [\n...     \">>fr<< this is a sentence in english that we want to translate to french\",\n...     \">>pt<< This should go to portuguese\",\n...     \">>es<< And this to Spanish\",\n... ]\n\n>>> model_name = \"Helsinki-NLP/opus-mt-en-ROMANCE\"\n>>> tokenizer = MarianTokenizer.from_pretrained(model_name)\n\n>>> model = MarianMTModel.from_pretrained(model_name)\n>>> translated = model.generate(**tokenizer(src_text, return_tensors=\"pt\", padding=True))\n>>> tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]\n[\"c'est une phrase en anglais que nous voulons traduire en fran\u00e7ais\", \n 'Isto deve ir para o portugu\u00eas.',\n 'Y esto al espa\u00f1ol']\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## MarianConfig\n\n[[autodoc]] MarianConfig\n\n## MarianTokenizer\n\n[[autodoc]] MarianTokenizer\n    - build_inputs_with_special_tokens\n\n<frameworkcontent>\n<pt>\n\n## MarianModel\n\n[[autodoc]] MarianModel\n    - forward\n\n## MarianMTModel\n\n[[autodoc]] MarianMTModel\n    - forward\n\n## MarianForCausalLM\n\n[[autodoc]] MarianForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFMarianModel\n\n[[autodoc]] TFMarianModel\n    - call\n\n## TFMarianMTModel\n\n[[autodoc]] TFMarianMTModel\n    - call\n\n</tf>\n<jax>\n\n## FlaxMarianModel\n\n[[autodoc]] FlaxMarianModel\n    - __call__\n\n## FlaxMarianMTModel\n\n[[autodoc]] FlaxMarianMTModel\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2146b1b6-0759-4a1a-8ad1-fb3dc684c7b9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vision-text-dual-encoder/", "title": "VisionTextDualEncoder", "tokens": 543, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2146b1b6-0759-4a1a-8ad1-fb3dc684c7b9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-text-dual-encoder/\", \"title\": \"VisionTextDualEncoder\", \"tokens\": 543, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b089cfeb-af35-46d7-ab6d-3a99496e51de\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-text-dual-encoder/\", \"title\": \"VisionTextDualEncoder\", \"tokens\": 543, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b242f89312a2c8b5841215ec6410d9e9a0f338958a2c9fed9df342174e57894b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1579, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b089cfeb-af35-46d7-ab6d-3a99496e51de", "doc_id": "b089cfeb-af35-46d7-ab6d-3a99496e51de", "ref_doc_id": "b089cfeb-af35-46d7-ab6d-3a99496e51de", "chroma:document": "# VisionTextDualEncoder\n\n## Overview\n\nThe [`VisionTextDualEncoderModel`] can be used to initialize a vision-text dual encoder model with\nany pretrained vision autoencoding model as the vision encoder (*e.g.* [ViT](vit), [BEiT](beit), [DeiT](deit)) and any pretrained text autoencoding model as the text encoder (*e.g.* [RoBERTa](roberta), [BERT](bert)). Two projection layers are added on top of both the vision and text encoder to project the output embeddings\nto a shared latent space. The projection layers are randomly initialized so the model should be fine-tuned on a\ndownstream task. This model can be used to align the vision-text embeddings using CLIP like contrastive image-text\ntraining and then can be used for zero-shot vision tasks such image-classification or retrieval.\n\nIn [LiT: Zero-Shot Transfer with Locked-image Text Tuning](https://arxiv.org/abs/2111.07991) it is shown how\nleveraging pre-trained (locked/frozen) image and text model for contrastive learning yields significant improvement on\nnew zero-shot vision tasks such as image classification or retrieval.\n\n## VisionTextDualEncoderConfig\n\n[[autodoc]] VisionTextDualEncoderConfig\n\n## VisionTextDualEncoderProcessor\n\n[[autodoc]] VisionTextDualEncoderProcessor\n\n<frameworkcontent>\n<pt>\n\n## VisionTextDualEncoderModel\n\n[[autodoc]] VisionTextDualEncoderModel\n    - forward\n\n</pt>\n<tf>\n\n## FlaxVisionTextDualEncoderModel\n\n[[autodoc]] FlaxVisionTextDualEncoderModel\n    - __call__\n\n</tf>\n<jax>\n\n## TFVisionTextDualEncoderModel\n\n[[autodoc]] TFVisionTextDualEncoderModel\n    - call\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cf36e9b2-e601-4633-a1f3-b79079bb25d8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cf36e9b2-e601-4633-a1f3-b79079bb25d8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f92b581-716a-4047-8f46-1fa6d7e256e6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a37bb9a3daf60c76130a430c85ceb53fb567c2c1575d836c4a74949e3a9697e6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"748c059e-f831-4e36-94bf-5a412230141e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2d00388786c48a4f235b7d2e610f12909bbbfd51ff605d8952ed50a732463673\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3248, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "doc_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "ref_doc_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "chroma:document": "# NLLB-MOE\n\n\n## Overview\n\nThe NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-juss\u00e0, James Cross, Onur \u00c7elebi,\nMaha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,\nLoic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,\nNecip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm\u00e1n, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,\nSafiyyah Saleem, Holger Schwenk, and Jeff Wang.\n\nThe abstract of the paper is the following:\n\n*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.\nHowever, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the\n200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind? In No Language Left Behind, we took on this challenge by\nfirst contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed\nat narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of\nExperts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training\nimprovements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using\na human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.\nOur model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/facebookresearch/fairseq).\n\n## Usage tips\n\n- M2M100ForConditionalGeneration is the base model for both NLLB and NLLB MoE\n- The NLLB-MoE is very similar to the NLLB model, but it's feed forward layer is based on the implementation of SwitchTransformers.\n- The tokenizer is the same as the NLLB models.\n\n## Implementation differences with SwitchTransformers\n\nThe biggest difference is the way the tokens are routed. NLLB-MoE uses a `top-2-gate` which means that for each input, only the top two experts are selected based on the \nhighest predicted probabilities from the gating network, and the remaining experts are ignored. In `SwitchTransformers`, only the top-1 probabilities are computed, \nwhich means that tokens have less probability of being forwarded."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 748c059e-f831-4e36-94bf-5a412230141e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"748c059e-f831-4e36-94bf-5a412230141e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f92b581-716a-4047-8f46-1fa6d7e256e6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a37bb9a3daf60c76130a430c85ceb53fb567c2c1575d836c4a74949e3a9697e6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cf36e9b2-e601-4633-a1f3-b79079bb25d8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"783ee40db1474b31e5b2cc41c13efd257e2d92d7d770f70e1557da5733348484\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a6739a23-44e1-45b4-9f5e-26ad20d9d80a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"385c780de5a64533d87c0100236afa3bd959ff47ba979011d2e4c1060ad2c756\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1457, \"end_char_idx\": 4426, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "doc_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "ref_doc_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "chroma:document": "Then, we created datasets and models aimed\nat narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of\nExperts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training\nimprovements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using\na human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.\nOur model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/facebookresearch/fairseq).\n\n## Usage tips\n\n- M2M100ForConditionalGeneration is the base model for both NLLB and NLLB MoE\n- The NLLB-MoE is very similar to the NLLB model, but it's feed forward layer is based on the implementation of SwitchTransformers.\n- The tokenizer is the same as the NLLB models.\n\n## Implementation differences with SwitchTransformers\n\nThe biggest difference is the way the tokens are routed. NLLB-MoE uses a `top-2-gate` which means that for each input, only the top two experts are selected based on the \nhighest predicted probabilities from the gating network, and the remaining experts are ignored. In `SwitchTransformers`, only the top-1 probabilities are computed, \nwhich means that tokens have less probability of being forwarded. Moreover, if a token is not routed to any expert, `SwitchTransformers` still adds its unmodified hidden \nstates (kind of like a residual connection) while they are masked in `NLLB`'s top-2 routing mechanism. \n\n## Generating with NLLB-MoE\n\nThe available checkpoints require around 350GB of storage. Make sure to use `accelerate` if you do not have enough RAM on your machine.\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-moe-54b\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-moe-54b\")\n\n>>> article = \"Previously, Ring's CEO, Jamie Siminoff, remarked the company started when his doorbell wasn't audible from his shop in his garage.\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a6739a23-44e1-45b4-9f5e-26ad20d9d80a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a6739a23-44e1-45b4-9f5e-26ad20d9d80a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f92b581-716a-4047-8f46-1fa6d7e256e6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a37bb9a3daf60c76130a430c85ceb53fb567c2c1575d836c4a74949e3a9697e6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"748c059e-f831-4e36-94bf-5a412230141e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c05da317b8ca57078db2d04d927f6b454194b7d397c63b00ee99afe0d471a061\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e2cb1e5b-c41f-4768-bd67-09569e8dd91c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f3759b436ce7e9a0576462292acf4dece303bc24ae4a08e06a544a7e6645a48\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2792, \"end_char_idx\": 5943, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "doc_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "ref_doc_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "chroma:document": "## Implementation differences with SwitchTransformers\n\nThe biggest difference is the way the tokens are routed. NLLB-MoE uses a `top-2-gate` which means that for each input, only the top two experts are selected based on the \nhighest predicted probabilities from the gating network, and the remaining experts are ignored. In `SwitchTransformers`, only the top-1 probabilities are computed, \nwhich means that tokens have less probability of being forwarded. Moreover, if a token is not routed to any expert, `SwitchTransformers` still adds its unmodified hidden \nstates (kind of like a residual connection) while they are masked in `NLLB`'s top-2 routing mechanism. \n\n## Generating with NLLB-MoE\n\nThe available checkpoints require around 350GB of storage. Make sure to use `accelerate` if you do not have enough RAM on your machine.\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-moe-54b\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-moe-54b\")\n\n>>> article = \"Previously, Ring's CEO, Jamie Siminoff, remarked the company started when his doorbell wasn't audible from his shop in his garage.\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"fra_Latn\"], max_length=50\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"Auparavant, le PDG de Ring, Jamie Siminoff, a fait remarquer que la soci\u00e9t\u00e9 avait commenc\u00e9 lorsque sa sonnette n'\u00e9tait pas audible depuis son magasin dans son garage.\"\n```\n\n### Generating from any other language than English\n\nEnglish (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.\n\nSee example below for a translation from romanian to german:\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-moe-54b\", src_lang=\"ron_Latn\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-moe-54b\")\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"deu_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e2cb1e5b-c41f-4768-bd67-09569e8dd91c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e2cb1e5b-c41f-4768-bd67-09569e8dd91c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f92b581-716a-4047-8f46-1fa6d7e256e6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a37bb9a3daf60c76130a430c85ceb53fb567c2c1575d836c4a74949e3a9697e6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a6739a23-44e1-45b4-9f5e-26ad20d9d80a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb-moe/\", \"title\": \"NLLB-MOE\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c7a8da6f91bd1faffa2b2d9691d922c30c6e837d1c556f6f504271f9221f8ab3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4427, \"end_char_idx\": 6292, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "doc_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "ref_doc_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "chroma:document": ">>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"fra_Latn\"], max_length=50\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"Auparavant, le PDG de Ring, Jamie Siminoff, a fait remarquer que la soci\u00e9t\u00e9 avait commenc\u00e9 lorsque sa sonnette n'\u00e9tait pas audible depuis son magasin dans son garage.\"\n```\n\n### Generating from any other language than English\n\nEnglish (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.\n\nSee example below for a translation from romanian to german:\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-moe-54b\", src_lang=\"ron_Latn\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-moe-54b\")\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"deu_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n\n## NllbMoeConfig\n\n[[autodoc]] NllbMoeConfig\n\n## NllbMoeTop2Router\n\n[[autodoc]] NllbMoeTop2Router\n    - route_tokens\n    - forward\n\n## NllbMoeSparseMLP\n\n[[autodoc]] NllbMoeSparseMLP\n    - forward\n\n## NllbMoeModel\n\n[[autodoc]] NllbMoeModel\n    - forward\n\n## NllbMoeForConditionalGeneration\n\n[[autodoc]] NllbMoeForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 20c7be2e-f7e3-42a8-94e4-1b9ecce77daa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"20c7be2e-f7e3-42a8-94e4-1b9ecce77daa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"65c2649f-0dcc-4be9-b3ce-d69c5288c992\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e0003566b09b76bb461a72f902c9cf83e4b9dad4380a2130fecafc6214ca2a00\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f115a5dd-8032-4326-aafa-97ee59374701\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e1e69a561b5a68cd927ec376477ee5d56399eefd95e3659b365cd237db801918\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2702, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "doc_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "ref_doc_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "chroma:document": "# GPTSAN-japanese\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe GPTSAN-japanese model was released in the repository by Toshiyuki Sakamoto (tanreinama).\n\nGPTSAN is a Japanese language model using Switch Transformer. It has the same structure as the model introduced as Prefix LM\nin the T5 paper, and support both Text Generation and Masked Language Modeling tasks. These basic tasks similarly can\nfine-tune for translation or summarization.\n\n### Usage example\n\nThe `generate()` method can be used to generate text using GPTSAN-Japanese model.\n\n```python\n>>> from transformers import AutoModel, AutoTokenizer\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Tanrei/GPTSAN-japanese\")\n>>> model = AutoModel.from_pretrained(\"Tanrei/GPTSAN-japanese\").cuda()\n>>> x_tok = tokenizer(\"\u306f\u3001\", prefix_text=\"\u7e54\u7530\u4fe1\u9577\", return_tensors=\"pt\")\n>>> torch.manual_seed(0)\n>>> gen_tok = model.generate(x_tok.input_ids.cuda(), token_type_ids=x_tok.token_type_ids.cuda(), max_new_tokens=20)\n>>> tokenizer.decode(gen_tok[0])\n'\u7e54\u7530\u4fe1\u9577\u306f\u30012004\u5e74\u306b\u300e\u6226\u56fdBASARA\u300f\u306e\u305f\u3081\u306b\u3001\u8c4a\u81e3\u79c0\u5409'\n```\n\n## GPTSAN Features\n\nGPTSAN has some unique features. It has a model structure of Prefix-LM. It works as a shifted Masked Language Model for Prefix Input tokens. Un-prefixed inputs behave like normal generative models.\nThe Spout vector is a GPTSAN specific input. Spout is pre-trained with random inputs, but you can specify a class of text or an arbitrary vector during fine-tuning. This allows you to indicate the tendency of the generated text.\nGPTSAN has a sparse Feed Forward based on Switch-Transformer. You can also add other layers and train them partially. See the original GPTSAN repository for details.\n\n### Prefix-LM Model\n\nGPTSAN has the structure of the model named Prefix-LM in the `T5` paper. (The original GPTSAN repository calls it `hybrid`)\nIn GPTSAN, the `Prefix` part of Prefix-LM, that is, the input position that can be referenced by both tokens, can be specified with any length.\nArbitrary lengths can also be specified differently for each batch.\nThis length applies to the text entered in `prefix_text` for the tokenizer.\nThe tokenizer returns the mask of the `Prefix` part of Prefix-LM as `token_type_ids`.\nThe model treats the part where `token_type_ids` is 1 as a `Prefix` part, that is, the input can refer to both tokens before and after.\n\n## Usage tips\n\nSpecifying the Prefix part is done with a mask passed to self-attention."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f115a5dd-8032-4326-aafa-97ee59374701
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f115a5dd-8032-4326-aafa-97ee59374701\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"65c2649f-0dcc-4be9-b3ce-d69c5288c992\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e0003566b09b76bb461a72f902c9cf83e4b9dad4380a2130fecafc6214ca2a00\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"20c7be2e-f7e3-42a8-94e4-1b9ecce77daa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bc4b08b4a3eaf147f779260cc67442ca53d32545b1542e157747f8a08746313b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"92bbbe8d-9658-45ca-a95b-dc2041e8d003\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"74a89b65a6c9a6dcf31a94c0d7d0c90b94e4a22402997d946631154377c4088c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1398, \"end_char_idx\": 3695, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "doc_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "ref_doc_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "chroma:document": "It has a model structure of Prefix-LM. It works as a shifted Masked Language Model for Prefix Input tokens. Un-prefixed inputs behave like normal generative models.\nThe Spout vector is a GPTSAN specific input. Spout is pre-trained with random inputs, but you can specify a class of text or an arbitrary vector during fine-tuning. This allows you to indicate the tendency of the generated text.\nGPTSAN has a sparse Feed Forward based on Switch-Transformer. You can also add other layers and train them partially. See the original GPTSAN repository for details.\n\n### Prefix-LM Model\n\nGPTSAN has the structure of the model named Prefix-LM in the `T5` paper. (The original GPTSAN repository calls it `hybrid`)\nIn GPTSAN, the `Prefix` part of Prefix-LM, that is, the input position that can be referenced by both tokens, can be specified with any length.\nArbitrary lengths can also be specified differently for each batch.\nThis length applies to the text entered in `prefix_text` for the tokenizer.\nThe tokenizer returns the mask of the `Prefix` part of Prefix-LM as `token_type_ids`.\nThe model treats the part where `token_type_ids` is 1 as a `Prefix` part, that is, the input can refer to both tokens before and after.\n\n## Usage tips\n\nSpecifying the Prefix part is done with a mask passed to self-attention.\nWhen token_type_ids=None or all zero, it is equivalent to regular causal mask\n\nfor example:\n\n>>> x_token = tokenizer(\"\uff71\uff72\uff73\uff74\")\ninput_ids:      | SOT | SEG | \uff71 | \uff72 | \uff73 | \uff74 |\ntoken_type_ids: | 1   | 0   | 0 | 0 | 0 | 0 |\nprefix_lm_mask:\nSOT | 1 0 0 0 0 0 |\nSEG | 1 1 0 0 0 0 |\n\uff71   | 1 1 1 0 0 0 |\n\uff72   | 1 1 1 1 0 0 |\n\uff73   | 1 1 1 1 1 0 |\n\uff74   | 1 1 1 1 1 1 |\n\n>>> x_token = tokenizer(\"\", prefix_text=\"\uff71\uff72\uff73\uff74\")\ninput_ids:      | SOT | \uff71 | \uff72 | \uff73 | \uff74 | SEG |\ntoken_type_ids: | 1   | 1 | 1 | 1 | 1 | 0  |\nprefix_lm_mask:\nSOT | 1 1 1 1 1 0 |\n\uff71   | 1 1 1 1 1 0 |\n\uff72   | 1 1 1 1 1 0 |\n\uff73   | 1 1 1 1 1 0 |\n\uff74   | 1 1 1 1 1 0 |\nSEG | 1 1 1 1 1 1 |\n\n>>> x_token = tokenizer(\"\uff73\uff74\", prefix_text=\"\uff71\uff72\")\ninput_ids:      | SOT | \uff71 | \uff72 | SEG | \uff73 | \uff74 |\ntoken_type_ids: | 1   | 1 | 1 | 0   | 0 | 0 |\nprefix_lm_mask:\nSOT | 1 1 1 0 0 0 |\n\uff71   | 1 1 1 0 0 0 |\n\uff72   | 1 1 1 0 0 0 |\nSEG | 1 1 1 1 0 0 |\n\uff73   | 1 1 1 1 1 0 |\n\uff74   | 1 1 1 1 1 1 |\n\n### Spout Vector\n\nA Spout Vector is a special vector for controlling text generation."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 92bbbe8d-9658-45ca-a95b-dc2041e8d003
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"92bbbe8d-9658-45ca-a95b-dc2041e8d003\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"65c2649f-0dcc-4be9-b3ce-d69c5288c992\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e0003566b09b76bb461a72f902c9cf83e4b9dad4380a2130fecafc6214ca2a00\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f115a5dd-8032-4326-aafa-97ee59374701\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/\", \"title\": \"GPTSAN-japanese\", \"tokens\": 1558, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"98e11795cb2b3716948b3433f08caa114c0dbff55807a53f1978b38bd24aeee9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3696, \"end_char_idx\": 4437, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "doc_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "ref_doc_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "chroma:document": "This vector is treated as the first embedding in self-attention to bring extraneous attention to the generated tokens.\nIn the pre-trained model published from `Tanrei/GPTSAN-japanese`, the Spout Vector is a 128-dimensional vector that passes through 8 fully connected layers in the model and is projected into the space acting as external attention.\nThe Spout Vector projected by the fully connected layer is split to be passed to all self-attentions.\n\n## GPTSanJapaneseConfig\n\n[[autodoc]] GPTSanJapaneseConfig\n\n## GPTSanJapaneseTokenizer\n\n[[autodoc]] GPTSanJapaneseTokenizer\n\n## GPTSanJapaneseModel\n\n[[autodoc]] GPTSanJapaneseModel\n\n## GPTSanJapaneseForConditionalGeneration\n\n[[autodoc]] GPTSanJapaneseForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 71fee20a-7fdc-4dcc-b9ff-c53b69edecba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nat/", "title": "Neighborhood Attention Transformer", "tokens": 1124, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"71fee20a-7fdc-4dcc-b9ff-c53b69edecba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat/\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e99cf0ee-37a9-470c-a7d6-de698838be2d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat/\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c5c40e9deff78bc8abcfdcbcd6450a7b1467e7270b619977c3e9e99a03002693\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f4730f8e-75f0-458f-a2a5-aa343379ac30\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d30e68a883caf4322ea4140dd9e352882be2759bc9b73a03abb8af50571ea3c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3159, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e99cf0ee-37a9-470c-a7d6-de698838be2d", "doc_id": "e99cf0ee-37a9-470c-a7d6-de698838be2d", "ref_doc_id": "e99cf0ee-37a9-470c-a7d6-de698838be2d", "chroma:document": "# Neighborhood Attention Transformer\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nNAT was proposed in [Neighborhood Attention Transformer](https://arxiv.org/abs/2204.07143)\nby Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi.\n\nIt is a hierarchical vision transformer based on Neighborhood Attention, a sliding-window self attention pattern.\n\nThe abstract from the paper is the following:\n\n*We present Neighborhood Attention (NA), the first efficient and scalable sliding-window attention mechanism for vision.\nNA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels, and therefore enjoys a\nlinear time and space complexity compared to the quadratic complexity of SA. The sliding-window pattern allows NA's\nreceptive field to grow without needing extra pixel shifts, and preserves translational equivariance, unlike\nSwin Transformer's Window Self Attention (WSA). We develop NATTEN (Neighborhood Attention Extension), a Python package\nwith efficient C++ and CUDA kernels, which allows NA to run up to 40% faster than Swin's WSA while using up to 25% less\nmemory. We further present Neighborhood Attention Transformer (NAT), a new hierarchical transformer design based on NA\nthat boosts image classification and downstream vision performance. Experimental results on NAT are competitive;\nNAT-Tiny reaches 83.2% top-1 accuracy on ImageNet, 51.4% mAP on MS-COCO and 48.4% mIoU on ADE20K, which is 1.9%\nImageNet accuracy, 1.0% COCO mAP, and 2.6% ADE20K mIoU improvement over a Swin model with similar size. *\n\n<img\nsrc=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/neighborhood-attention-pattern.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Neighborhood Attention compared to other attention patterns.\nTaken from the <a href=\"https://arxiv.org/abs/2204.07143\">original paper</a>.</small>\n\nThis model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).\nThe original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).\n\n## Usage tips\n\n- One can use the [`AutoImageProcessor`] API to prepare images for the model.\n- NAT can be used as a *backbone*. When `output_hidden_states = True`,\nit will output both `hidden_states` and `reshaped_hidden_states`.\nThe `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than\n`(batch_size, height, width, num_channels)`.\n\nNotes:\n- NAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention.\nYou can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten),\nor build on your system by running `pip install natten`.\nNote that the latter will likely take time to compile. NATTEN does not support Windows devices yet.\n- Patch size of 4 is only supported at the moment."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f4730f8e-75f0-458f-a2a5-aa343379ac30
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nat/", "title": "Neighborhood Attention Transformer", "tokens": 1124, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f4730f8e-75f0-458f-a2a5-aa343379ac30\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat/\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e99cf0ee-37a9-470c-a7d6-de698838be2d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat/\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c5c40e9deff78bc8abcfdcbcd6450a7b1467e7270b619977c3e9e99a03002693\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"71fee20a-7fdc-4dcc-b9ff-c53b69edecba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nat/\", \"title\": \"Neighborhood Attention Transformer\", \"tokens\": 1124, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0ea7a7acef97b69fe28f9b3f02f2d154e4063b1684c5ba8d2149067bad7974c6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1849, \"end_char_idx\": 4093, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e99cf0ee-37a9-470c-a7d6-de698838be2d", "doc_id": "e99cf0ee-37a9-470c-a7d6-de698838be2d", "ref_doc_id": "e99cf0ee-37a9-470c-a7d6-de698838be2d", "chroma:document": "*\n\n<img\nsrc=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/neighborhood-attention-pattern.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Neighborhood Attention compared to other attention patterns.\nTaken from the <a href=\"https://arxiv.org/abs/2204.07143\">original paper</a>.</small>\n\nThis model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).\nThe original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).\n\n## Usage tips\n\n- One can use the [`AutoImageProcessor`] API to prepare images for the model.\n- NAT can be used as a *backbone*. When `output_hidden_states = True`,\nit will output both `hidden_states` and `reshaped_hidden_states`.\nThe `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than\n`(batch_size, height, width, num_channels)`.\n\nNotes:\n- NAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention.\nYou can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten),\nor build on your system by running `pip install natten`.\nNote that the latter will likely take time to compile. NATTEN does not support Windows devices yet.\n- Patch size of 4 is only supported at the moment.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with NAT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`NatForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## NatConfig\n\n[[autodoc]] NatConfig\n\n## NatModel\n\n[[autodoc]] NatModel\n    - forward\n\n## NatForImageClassification\n\n[[autodoc]] NatForImageClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a5b89ce6-b69e-46e2-ac8d-9c969c53b9c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a5b89ce6-b69e-46e2-ac8d-9c969c53b9c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cdece756-509c-426d-b48a-b086a295b00c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a7534d64910fb25dc679e8a2abc0a3bdf49eeb539b4efaa54583429adf616634\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"208ea302-a92b-4a81-9380-99e43747148e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3a1931a4ab0ad52105f36450058ff76de69a3bc0de520be2f82a5aeb0f9dc149\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2191, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cdece756-509c-426d-b48a-b086a295b00c", "doc_id": "cdece756-509c-426d-b48a-b086a295b00c", "ref_doc_id": "cdece756-509c-426d-b48a-b086a295b00c", "chroma:document": "# Depth Anything V2\n\n## Overview\n\nDepth Anything V2 was introduced in [the paper of the same name](https://arxiv.org/abs/2406.09414) by Lihe Yang et al. It uses the same architecture as the original [Depth Anything model](depth_anything), but uses synthetic data and a larger capacity teacher model to achieve much finer and robust depth predictions.\n\nThe abstract from the paper is the following:\n\n*This work presents Depth Anything V2. Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more robust depth predictions through three key practices: 1) replacing all labeled real images with synthetic images, 2) scaling up the capacity of our teacher model, and 3) teaching student models via the bridge of large-scale pseudo-labeled real images. Compared with the latest models built on Stable Diffusion, our models are significantly more efficient (more than 10x faster) and more accurate. We offer models of different scales (ranging from 25M to 1.3B params) to support extensive scenarios. Benefiting from their strong generalization capability, we fine-tune them with metric depth labels to obtain our metric depth models. In addition to our models, considering the limited diversity and frequent noise in current test sets, we construct a versatile evaluation benchmark with precise annotations and diverse scenes to facilitate future research.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Depth Anything overview. Taken from the <a href=\"https://arxiv.org/abs/2401.10891\">original paper</a>.</small>\n\nThe Depth Anything models were contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/DepthAnything/Depth-Anything-V2).\n\n## Usage example\n\nThere are 2 main ways to use Depth Anything V2: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 208ea302-a92b-4a81-9380-99e43747148e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"208ea302-a92b-4a81-9380-99e43747148e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cdece756-509c-426d-b48a-b086a295b00c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a7534d64910fb25dc679e8a2abc0a3bdf49eeb539b4efaa54583429adf616634\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a5b89ce6-b69e-46e2-ac8d-9c969c53b9c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87cc6e947a538a994b9457976f50ad4a8c311ef7510a8748051f5d319ca6c5bf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"09d454a1-27c9-43de-baaa-6de06d9bc27f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e9c07136b9189b0ce9b5900b21ed8233444d62bb311fc64bd3b68152e377a8a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 438, \"end_char_idx\": 4037, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cdece756-509c-426d-b48a-b086a295b00c", "doc_id": "cdece756-509c-426d-b48a-b086a295b00c", "ref_doc_id": "cdece756-509c-426d-b48a-b086a295b00c", "chroma:document": "Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more robust depth predictions through three key practices: 1) replacing all labeled real images with synthetic images, 2) scaling up the capacity of our teacher model, and 3) teaching student models via the bridge of large-scale pseudo-labeled real images. Compared with the latest models built on Stable Diffusion, our models are significantly more efficient (more than 10x faster) and more accurate. We offer models of different scales (ranging from 25M to 1.3B params) to support extensive scenarios. Benefiting from their strong generalization capability, we fine-tune them with metric depth labels to obtain our metric depth models. In addition to our models, considering the limited diversity and frequent noise in current test sets, we construct a versatile evaluation benchmark with precise annotations and diverse scenes to facilitate future research.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Depth Anything overview. Taken from the <a href=\"https://arxiv.org/abs/2401.10891\">original paper</a>.</small>\n\nThe Depth Anything models were contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/DepthAnything/Depth-Anything-V2).\n\n## Usage example\n\nThere are 2 main ways to use Depth Anything V2: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself.\n\n### Pipeline API\n\nThe pipeline allows to use the model in a few lines of code:\n\n```python\n>>> from transformers import pipeline\n>>> from PIL import Image\n>>> import requests\n\n>>> # load pipe\n>>> pipe = pipeline(task=\"depth-estimation\", model=\"depth-anything/Depth-Anything-V2-Small-hf\")\n\n>>> # load image\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> # inference\n>>> depth = pipe(image)[\"depth\"]\n```\n\n### Using the model yourself\n\nIf you want to do the pre- and post-processing yourself, here's how to do that:\n\n```python\n>>> from transformers import AutoImageProcessor, AutoModelForDepthEstimation\n>>> import torch\n>>> import numpy as np\n>>> from PIL import Image\n>>> import requests\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\"depth-anything/Depth-Anything-V2-Small-hf\")\n>>> model = AutoModelForDepthEstimation.from_pretrained(\"depth-anything/Depth-Anything-V2-Small-hf\")\n\n>>> # prepare image for the model\n>>> inputs = image_processor(images=image, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n...     predicted_depth = outputs.predicted_depth\n\n>>> # interpolate to original size\n>>> prediction = torch.nn.functional.interpolate(\n...     predicted_depth.unsqueeze(1),\n...     size=image.size[::-1],\n...     mode=\"bicubic\",\n...     align_corners=False,\n... )\n\n>>> # visualize the prediction\n>>> output = prediction.squeeze().cpu().numpy()\n>>> formatted = (output * 255 / np.max(output)).astype(\"uint8\")\n>>> depth = Image.fromarray(formatted)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Depth Anything."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 09d454a1-27c9-43de-baaa-6de06d9bc27f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"09d454a1-27c9-43de-baaa-6de06d9bc27f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cdece756-509c-426d-b48a-b086a295b00c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a7534d64910fb25dc679e8a2abc0a3bdf49eeb539b4efaa54583429adf616634\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"208ea302-a92b-4a81-9380-99e43747148e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/\", \"title\": \"Depth Anything V2\", \"tokens\": 1299, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e8d227c962314472402aedddcf385a321a037b100b87f26f20708a0a1a87f78b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4039, \"end_char_idx\": 4948, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cdece756-509c-426d-b48a-b086a295b00c", "doc_id": "cdece756-509c-426d-b48a-b086a295b00c", "ref_doc_id": "cdece756-509c-426d-b48a-b086a295b00c", "chroma:document": "- [Monocular depth estimation task guide](../tasks/depth_estimation)\n- [Depth Anything V2 demo](https://huggingface.co/spaces/depth-anything/Depth-Anything-V2).\n- A notebook showcasing inference with [`DepthAnythingForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Depth%20Anything/Predicting_depth_in_an_image_with_Depth_Anything.ipynb). \ud83c\udf0e\n- [Core ML conversion of the `small` variant for use on Apple Silicon](https://huggingface.co/apple/coreml-depth-anything-v2-small).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DepthAnythingConfig\n\n[[autodoc]] DepthAnythingConfig\n\n## DepthAnythingForDepthEstimation\n\n[[autodoc]] DepthAnythingForDepthEstimation\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e8934f6f-3459-4b7f-bebe-37606ddd604c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e8934f6f-3459-4b7f-bebe-37606ddd604c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d1c30aed-e608-4f55-ac98-da2ab71a54c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"27c54315-fc4d-42c7-afc8-222f7f71dd1b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"22770f56cc9796afcdbcaae6a9181fedc64997e178a6675226f432c5413d9aa4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3419, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "ref_doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "chroma:document": "# ALBERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=albert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-albert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/albert-base-v2\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe ALBERT model was proposed in [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942) by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma,\nRadu Soricut. It presents two parameter-reduction techniques to lower memory consumption and increase the training\nspeed of BERT:\n\n- Splitting the embedding matrix into two smaller matrices.\n- Using repeating layers split among groups.\n\nThe abstract from the paper is the following:\n\n*Increasing model size when pretraining natural language representations often results in improved performance on\ndownstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations,\nlonger training times, and unexpected model degradation. To address these problems, we present two parameter-reduction\ntechniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows\nthat our proposed methods lead to models that scale much better compared to the original BERT. We also use a\nself-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks\nwith multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and\nSQuAD benchmarks while having fewer parameters compared to BERT-large.*\n\nThis model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by\n[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).\n\n## Usage tips\n\n- ALBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather\n  than the left.\n- ALBERT uses repeating layers which results in a small memory footprint, however the computational cost remains\n  similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same\n  number of (repeating) layers.\n- Embedding size E is different from hidden size H justified because the embeddings are context independent (one embedding vector represents one token), whereas hidden states are context dependent (one hidden state represents a sequence of tokens) so it's more logical to have H >> E. Also, the embedding matrix is large since it's V x E (V being the vocab size). If E < H, it has less parameters.\n- Layers are split in groups that share parameters (to save memory).\nNext sentence prediction is replaced by a sentence ordering prediction: in the inputs, we have two sentences A and B (that are consecutive) and we either feed A followed by B or B followed by A. The model must predict if they have been swapped or not.\n\n\n\nThis model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by\n[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).\n\n\n## Resources"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 27c54315-fc4d-42c7-afc8-222f7f71dd1b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"27c54315-fc4d-42c7-afc8-222f7f71dd1b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d1c30aed-e608-4f55-ac98-da2ab71a54c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e8934f6f-3459-4b7f-bebe-37606ddd604c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"05c9d855cec923846575ddb5535c20fe9d06c2ce9d8d2bb381820cee229c4a09\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"314949c2-03c8-4ff5-9d64-e296d92c5c6e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3313f2db2e48fc4ccefa876670b462a06683db2724d3bc03cf6d0da0c79e8619\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3159, \"end_char_idx\": 6491, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "ref_doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "chroma:document": "This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by\n[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).\n\n\n## Resources\n\n\nThe resources provided in the following sections consist of a list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with AlBERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n\n- [`AlbertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification).\n\n\n- [`TFAlbertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification).\n\n- [`FlaxAlbertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- Check the [Text classification task guide](../tasks/sequence_classification) on how to use the model.\n\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n\n- [`AlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification).\n\n\n- [`TFAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n\n\n\n- [`FlaxAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Token classification task guide](../tasks/token_classification) on how to use the model.\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`AlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 314949c2-03c8-4ff5-9d64-e296d92c5c6e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"314949c2-03c8-4ff5-9d64-e296d92c5c6e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d1c30aed-e608-4f55-ac98-da2ab71a54c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"27c54315-fc4d-42c7-afc8-222f7f71dd1b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6360e5d37b11bb2eaf6c4b95338eef0f84befb4a0e92247255dba64ae8c5b146\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"12fb498d-2da7-4e2f-ae6f-6e7f4d763b07\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9c5b42ddac3b7a3e3e0ac1ebc0874e038544a2735244f9adbb3832ba708c9736\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4838, \"end_char_idx\": 8013, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "ref_doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "chroma:document": "- [`TFAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n\n\n\n- [`FlaxAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Token classification task guide](../tasks/token_classification) on how to use the model.\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`AlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Masked language modeling task guide](../tasks/masked_language_modeling) on how to use the model.\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`AlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Question answering task guide](../tasks/question_answering) on how to use the model.\n\n**Multiple choice**\n\n- [`AlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 12fb498d-2da7-4e2f-ae6f-6e7f4d763b07
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"12fb498d-2da7-4e2f-ae6f-6e7f4d763b07\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d1c30aed-e608-4f55-ac98-da2ab71a54c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"314949c2-03c8-4ff5-9d64-e296d92c5c6e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"de823b9d7c474be8386dcfc63aff6021bb8ea0ccad4f79ca2f01e4963f692c4d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4300a835-6462-4a76-9d66-fdfd68dc985f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"816c4fae17cd9a3519904b5ec9d38fa0aaded8d39b3a74e03e0a39e09cb99646\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6492, \"end_char_idx\": 8392, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "ref_doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "chroma:document": "- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Masked language modeling task guide](../tasks/masked_language_modeling) on how to use the model.\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`AlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Question answering task guide](../tasks/question_answering) on how to use the model.\n\n**Multiple choice**\n\n- [`AlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFAlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n\n- Check the  [Multiple choice task guide](../tasks/multiple_choice) on how to use the model."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4300a835-6462-4a76-9d66-fdfd68dc985f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4300a835-6462-4a76-9d66-fdfd68dc985f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d1c30aed-e608-4f55-ac98-da2ab71a54c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"12fb498d-2da7-4e2f-ae6f-6e7f4d763b07\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/albert/\", \"title\": \"ALBERT\", \"tokens\": 2698, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"28376fb110c627aab1112bb713b93ad5b2e1a8377235dce085e6a9f555374967\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7050, \"end_char_idx\": 10613, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "ref_doc_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "chroma:document": "- [`TFAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- Check the [Question answering task guide](../tasks/question_answering) on how to use the model.\n\n**Multiple choice**\n\n- [`AlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFAlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n\n- Check the  [Multiple choice task guide](../tasks/multiple_choice) on how to use the model.\n\n\n## AlbertConfig\n\n[[autodoc]] AlbertConfig\n\n## AlbertTokenizer\n\n[[autodoc]] AlbertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## AlbertTokenizerFast\n\n[[autodoc]] AlbertTokenizerFast\n\n## Albert specific outputs\n\n[[autodoc]] models.albert.modeling_albert.AlbertForPreTrainingOutput\n\n[[autodoc]] models.albert.modeling_tf_albert.TFAlbertForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## AlbertModel\n\n[[autodoc]] AlbertModel\n    - forward\n\n## AlbertForPreTraining\n\n[[autodoc]] AlbertForPreTraining\n    - forward\n\n## AlbertForMaskedLM\n\n[[autodoc]] AlbertForMaskedLM\n    - forward\n\n## AlbertForSequenceClassification\n\n[[autodoc]] AlbertForSequenceClassification\n    - forward\n\n## AlbertForMultipleChoice\n\n[[autodoc]] AlbertForMultipleChoice\n\n## AlbertForTokenClassification\n\n[[autodoc]] AlbertForTokenClassification\n    - forward\n\n## AlbertForQuestionAnswering\n\n[[autodoc]] AlbertForQuestionAnswering\n    - forward\n\n</pt>\n\n<tf>\n\n## TFAlbertModel\n\n[[autodoc]] TFAlbertModel\n    - call\n\n## TFAlbertForPreTraining\n\n[[autodoc]] TFAlbertForPreTraining\n    - call\n\n## TFAlbertForMaskedLM\n\n[[autodoc]] TFAlbertForMaskedLM\n    - call\n\n## TFAlbertForSequenceClassification\n\n[[autodoc]] TFAlbertForSequenceClassification\n    - call\n\n## TFAlbertForMultipleChoice\n\n[[autodoc]] TFAlbertForMultipleChoice\n    - call\n\n## TFAlbertForTokenClassification\n\n[[autodoc]] TFAlbertForTokenClassification\n    - call\n\n## TFAlbertForQuestionAnswering\n\n[[autodoc]] TFAlbertForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxAlbertModel\n\n[[autodoc]] FlaxAlbertModel\n    - __call__\n\n## FlaxAlbertForPreTraining\n\n[[autodoc]] FlaxAlbertForPreTraining\n    - __call__\n\n## FlaxAlbertForMaskedLM\n\n[[autodoc]] FlaxAlbertForMaskedLM\n    - __call__\n\n## FlaxAlbertForSequenceClassification\n\n[[autodoc]] FlaxAlbertForSequenceClassification\n    - __call__\n\n## FlaxAlbertForMultipleChoice\n\n[[autodoc]] FlaxAlbertForMultipleChoice\n    - __call__\n\n## FlaxAlbertForTokenClassification\n\n[[autodoc]] FlaxAlbertForTokenClassification\n    - __call__\n\n## FlaxAlbertForQuestionAnswering\n\n[[autodoc]] FlaxAlbertForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6e537342-8a77-4684-b21e-212832b57fe8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vitdet/", "title": "ViTDet", "tokens": 512, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6e537342-8a77-4684-b21e-212832b57fe8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vitdet/\", \"title\": \"ViTDet\", \"tokens\": 512, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"813d82b3-adbf-4c9d-9f43-ae7b8cb76271\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vitdet/\", \"title\": \"ViTDet\", \"tokens\": 512, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e2b431e2f1f206f1aef277136745dbcdb0ca80cfa165644069936ce0c9862d32\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1674, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "813d82b3-adbf-4c9d-9f43-ae7b8cb76271", "doc_id": "813d82b3-adbf-4c9d-9f43-ae7b8cb76271", "ref_doc_id": "813d82b3-adbf-4c9d-9f43-ae7b8cb76271", "chroma:document": "# ViTDet\n\n## Overview\n\nThe ViTDet model was proposed in [Exploring Plain Vision Transformer Backbones for Object Detection](https://arxiv.org/abs/2203.16527) by Yanghao Li, Hanzi Mao, Ross Girshick, Kaiming He.\nVitDet leverages the plain [Vision Transformer](vit) for the task of object detection.\n\nThe abstract from the paper is the following:\n\n*We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone network for object detection. This design enables the original ViT architecture to be fine-tuned for object detection without needing to redesign a hierarchical backbone for pre-training. With minimal adaptations for fine-tuning, our plain-backbone detector can achieve competitive results. Surprisingly, we observe: (i) it is sufficient to build a simple feature pyramid from a single-scale feature map (without the common FPN design) and (ii) it is sufficient to use window attention (without shifting) aided with very few cross-window propagation blocks. With plain ViT backbones pre-trained as Masked Autoencoders (MAE), our detector, named ViTDet, can compete with the previous leading methods that were all based on hierarchical backbones, reaching up to 61.3 AP_box on the COCO dataset using only ImageNet-1K pre-training. We hope our study will draw attention to research on plain-backbone detectors.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet).\n\nTips:\n\n- At the moment, only the backbone is available.\n\n## VitDetConfig\n\n[[autodoc]] VitDetConfig\n\n## VitDetModel\n\n[[autodoc]] VitDetModel\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: abe1fcc1-f79f-4ce1-ad42-20173649c0e8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text/", "title": "Speech2Text", "tokens": 1503, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"abe1fcc1-f79f-4ce1-ad42-20173649c0e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text/\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fdd5b2d7-8daa-4211-a450-64b6c79d1297\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text/\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4a4b989d63136dcc10e1dfea949c79271c58e33950178b5a4df1b3dd1360cebd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fb385bb3-4787-4b4f-9411-ccead91df85d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1aa50d67f9bdc367a17b930ac1561e6255583ee33e52e4d04aae6e0a228b9819\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2582, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fdd5b2d7-8daa-4211-a450-64b6c79d1297", "doc_id": "fdd5b2d7-8daa-4211-a450-64b6c79d1297", "ref_doc_id": "fdd5b2d7-8daa-4211-a450-64b6c79d1297", "chroma:document": "# Speech2Text\n\n## Overview\n\nThe Speech2Text model was proposed in [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://arxiv.org/abs/2010.05171) by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino. It's a\ntransformer-based seq2seq (encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech\nTranslation (ST). It uses a convolutional downsampler to reduce the length of speech inputs by 3/4th before they are\nfed into the encoder. The model is trained with standard autoregressive cross-entropy loss and generates the\ntranscripts/translations autoregressively. Speech2Text has been fine-tuned on several datasets for ASR and ST:\n[LibriSpeech](http://www.openslr.org/12), [CoVoST 2](https://github.com/facebookresearch/covost), [MuST-C](https://ict.fbk.eu/must-c/).\n\nThis model was contributed by [valhalla](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text).\n\n## Inference\n\nSpeech2Text is a speech model that accepts a float tensor of log-mel filter-bank features extracted from the speech\nsignal. It's a transformer-based seq2seq model, so the transcripts/translations are generated autoregressively. The\n`generate()` method can be used for inference.\n\nThe [`Speech2TextFeatureExtractor`] class is responsible for extracting the log-mel filter-bank\nfeatures. The [`Speech2TextProcessor`] wraps [`Speech2TextFeatureExtractor`] and\n[`Speech2TextTokenizer`] into a single instance to both extract the input features and decode the\npredicted token ids.\n\nThe feature extractor depends on `torchaudio` and the tokenizer depends on `sentencepiece` so be sure to\ninstall those packages before running the examples. You could either install those as extra speech dependencies with\n`pip install transformers\"[speech, sentencepiece]\"` or install the packages separately with `pip install torchaudio sentencepiece`. Also `torchaudio` requires the development version of the [libsndfile](http://www.mega-nerd.com/libsndfile/) package which can be installed via a system package manager. On Ubuntu it can\nbe installed as follows: `apt install libsndfile1-dev`\n\n- ASR and Speech Translation\n\n```python\n>>> import torch\n>>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> model = Speech2TextForConditionalGeneration.from_pretrained(\"facebook/s2t-small-librispeech-asr\")\n>>> processor = Speech2TextProcessor.from_pretrained(\"facebook/s2t-small-librispeech-asr\")"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fb385bb3-4787-4b4f-9411-ccead91df85d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text/", "title": "Speech2Text", "tokens": 1503, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fb385bb3-4787-4b4f-9411-ccead91df85d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text/\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fdd5b2d7-8daa-4211-a450-64b6c79d1297\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text/\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4a4b989d63136dcc10e1dfea949c79271c58e33950178b5a4df1b3dd1360cebd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"abe1fcc1-f79f-4ce1-ad42-20173649c0e8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text/\", \"title\": \"Speech2Text\", \"tokens\": 1503, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a6e95072223f2fea15488be98a29faf7f7e07b4beca0cfd0afd9f101c8846940\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2585, \"end_char_idx\": 5575, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fdd5b2d7-8daa-4211-a450-64b6c79d1297", "doc_id": "fdd5b2d7-8daa-4211-a450-64b6c79d1297", "ref_doc_id": "fdd5b2d7-8daa-4211-a450-64b6c79d1297", "chroma:document": ">>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_demo\", \"clean\", split=\"validation\")\n\n>>> inputs = processor(ds[0][\"audio\"][\"array\"], sampling_rate=ds[0][\"audio\"][\"sampling_rate\"], return_tensors=\"pt\")\n>>> generated_ids = model.generate(inputs[\"input_features\"], attention_mask=inputs[\"attention_mask\"])\n\n>>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> transcription\n['mister quilter is the apostle of the middle classes and we are glad to welcome his gospel']\n```\n\n- Multilingual speech translation\n\n  For multilingual speech translation models, `eos_token_id` is used as the `decoder_start_token_id` and\n  the target language id is forced as the first generated token. To force the target language id as the first\n  generated token, pass the `forced_bos_token_id` parameter to the `generate()` method. The following\n  example shows how to transate English speech to French text using the *facebook/s2t-medium-mustc-multilingual-st*\n  checkpoint.\n\n```python\n>>> import torch\n>>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> model = Speech2TextForConditionalGeneration.from_pretrained(\"facebook/s2t-medium-mustc-multilingual-st\")\n>>> processor = Speech2TextProcessor.from_pretrained(\"facebook/s2t-medium-mustc-multilingual-st\")\n\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_demo\", \"clean\", split=\"validation\")\n\n>>> inputs = processor(ds[0][\"audio\"][\"array\"], sampling_rate=ds[0][\"audio\"][\"sampling_rate\"], return_tensors=\"pt\")\n>>> generated_ids = model.generate(\n...     inputs[\"input_features\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     forced_bos_token_id=processor.tokenizer.lang_code_to_id[\"fr\"],\n... )\n\n>>> translation = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> translation\n[\"(Vid\u00e9o) Si M. Kilder est l'apossible des classes moyennes, et nous sommes heureux d'\u00eatre accueillis dans son \u00e9vangile.\"]\n```\n\nSee the [model hub](https://huggingface.co/models?filter=speech_to_text) to look for Speech2Text checkpoints.\n\n## Speech2TextConfig\n\n[[autodoc]] Speech2TextConfig\n\n## Speech2TextTokenizer\n\n[[autodoc]] Speech2TextTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## Speech2TextFeatureExtractor\n\n[[autodoc]] Speech2TextFeatureExtractor\n    - __call__\n\n## Speech2TextProcessor\n\n[[autodoc]] Speech2TextProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n<frameworkcontent>\n<pt>\n\n## Speech2TextModel\n\n[[autodoc]] Speech2TextModel\n    - forward\n\n## Speech2TextForConditionalGeneration\n\n[[autodoc]] Speech2TextForConditionalGeneration\n    - forward\n\n</pt>\n<tf>\n\n## TFSpeech2TextModel\n\n[[autodoc]] TFSpeech2TextModel\n    - call\n\n## TFSpeech2TextForConditionalGeneration\n\n[[autodoc]] TFSpeech2TextForConditionalGeneration\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c27773de-b80a-4f67-b2b6-508c42b1d2db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/autoformer/", "title": "Autoformer", "tokens": 739, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c27773de-b80a-4f67-b2b6-508c42b1d2db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/autoformer/\", \"title\": \"Autoformer\", \"tokens\": 739, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"da9cf386-6db3-4b70-82b6-2c810c3d6642\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/autoformer/\", \"title\": \"Autoformer\", \"tokens\": 739, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4d06474b6b9d7816dd4c9baf1b6120eb5baa8bd401499090073e7e4be39f2505\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2844, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "da9cf386-6db3-4b70-82b6-2c810c3d6642", "doc_id": "da9cf386-6db3-4b70-82b6-2c810c3d6642", "ref_doc_id": "da9cf386-6db3-4b70-82b6-2c810c3d6642", "chroma:document": "# Autoformer\n\n## Overview\n\nThe Autoformer model was proposed in [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://arxiv.org/abs/2106.13008) by Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long.\n\nThis model augments the Transformer as a deep decomposition architecture, which can progressively decompose the trend and seasonal components during the forecasting process.\n\nThe abstract from the paper is the following:\n\n*Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease.*\n\nThis model was contributed by [elisim](https://huggingface.co/elisim) and [kashif](https://huggingface.co/kashif).\nThe original code can be found [here](https://github.com/thuml/Autoformer).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Check out the Autoformer blog-post in HuggingFace blog: [Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer)](https://huggingface.co/blog/autoformer)\n\n## AutoformerConfig\n\n[[autodoc]] AutoformerConfig\n\n## AutoformerModel\n\n[[autodoc]] AutoformerModel\n    - forward\n\n## AutoformerForPrediction\n\n[[autodoc]] AutoformerForPrediction\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 447d28d6-d214-484f-81ac-032db874c0c3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clipseg/", "title": "CLIPSeg", "tokens": 1043, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"447d28d6-d214-484f-81ac-032db874c0c3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg/\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fdf89222-d043-4613-933e-b8cf6d668dc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg/\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9504ab89a5a8093c7de94bdbcdfe3221e6de0d71eb5c275539bdeee553d0ec8a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f5c964fc-de74-48d7-9db8-f659f74bc11c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69b59b69de3d8aabb21e7fa6aae36e15c30419d1b5a6e1a3f983bdd2132ed05d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3171, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fdf89222-d043-4613-933e-b8cf6d668dc8", "doc_id": "fdf89222-d043-4613-933e-b8cf6d668dc8", "ref_doc_id": "fdf89222-d043-4613-933e-b8cf6d668dc8", "chroma:document": "# CLIPSeg\n\n## Overview\n\nThe CLIPSeg model was proposed in [Image Segmentation Using Text and Image Prompts](https://arxiv.org/abs/2112.10003) by Timo L\u00fcddecke\nand Alexander Ecker. CLIPSeg adds a minimal decoder on top of a frozen [CLIP](clip) model for zero- and one-shot image segmentation.\n\nThe abstract from the paper is the following:\n\n*Image segmentation is usually addressed by training a\nmodel for a fixed set of object classes. Incorporating additional classes or more complex queries later is expensive\nas it requires re-training the model on a dataset that encompasses these expressions. Here we propose a system\nthat can generate image segmentations based on arbitrary\nprompts at test time. A prompt can be either a text or an\nimage. This approach enables us to create a unified model\n(trained once) for three common segmentation tasks, which\ncome with distinct challenges: referring expression segmentation, zero-shot segmentation and one-shot segmentation.\nWe build upon the CLIP model as a backbone which we extend with a transformer-based decoder that enables dense\nprediction. After training on an extended version of the\nPhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on\nan additional image expressing the query. We analyze different variants of the latter image-based prompts in detail.\nThis novel hybrid input allows for dynamic adaptation not\nonly to the three segmentation tasks mentioned above, but\nto any binary segmentation task where a text or image query\ncan be formulated. Finally, we find our system to adapt well\nto generalized queries involving affordances or properties*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/clipseg_architecture.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> CLIPSeg overview. Taken from the <a href=\"https://arxiv.org/abs/2112.10003\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/timojl/clipseg).\n\n## Usage tips\n\n- [`CLIPSegForImageSegmentation`] adds a decoder on top of [`CLIPSegModel`]. The latter is identical to [`CLIPModel`].\n- [`CLIPSegForImageSegmentation`] can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text\n(provided to the model as `input_ids`) or an image (provided to the model as `conditional_pixel_values`). One can also provide custom\nconditional embeddings (provided to the model as `conditional_embeddings`).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CLIPSeg. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"image-segmentation\"/>\n\n- A notebook that illustrates [zero-shot image segmentation with CLIPSeg](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/CLIPSeg/Zero_shot_image_segmentation_with_CLIPSeg.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f5c964fc-de74-48d7-9db8-f659f74bc11c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clipseg/", "title": "CLIPSeg", "tokens": 1043, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f5c964fc-de74-48d7-9db8-f659f74bc11c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg/\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fdf89222-d043-4613-933e-b8cf6d668dc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg/\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9504ab89a5a8093c7de94bdbcdfe3221e6de0d71eb5c275539bdeee553d0ec8a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"447d28d6-d214-484f-81ac-032db874c0c3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clipseg/\", \"title\": \"CLIPSeg\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"32d3c53f031d742afd16ac421a1bc9c1cf9d121cd765bee0ea6a8da17242c6b4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1563, \"end_char_idx\": 3735, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fdf89222-d043-4613-933e-b8cf6d668dc8", "doc_id": "fdf89222-d043-4613-933e-b8cf6d668dc8", "ref_doc_id": "fdf89222-d043-4613-933e-b8cf6d668dc8", "chroma:document": "Finally, we find our system to adapt well\nto generalized queries involving affordances or properties*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/clipseg_architecture.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> CLIPSeg overview. Taken from the <a href=\"https://arxiv.org/abs/2112.10003\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/timojl/clipseg).\n\n## Usage tips\n\n- [`CLIPSegForImageSegmentation`] adds a decoder on top of [`CLIPSegModel`]. The latter is identical to [`CLIPModel`].\n- [`CLIPSegForImageSegmentation`] can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text\n(provided to the model as `input_ids`) or an image (provided to the model as `conditional_pixel_values`). One can also provide custom\nconditional embeddings (provided to the model as `conditional_embeddings`).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CLIPSeg. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"image-segmentation\"/>\n\n- A notebook that illustrates [zero-shot image segmentation with CLIPSeg](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/CLIPSeg/Zero_shot_image_segmentation_with_CLIPSeg.ipynb).\n\n## CLIPSegConfig\n\n[[autodoc]] CLIPSegConfig\n    - from_text_vision_configs\n\n## CLIPSegTextConfig\n\n[[autodoc]] CLIPSegTextConfig\n\n## CLIPSegVisionConfig\n\n[[autodoc]] CLIPSegVisionConfig\n\n## CLIPSegProcessor\n\n[[autodoc]] CLIPSegProcessor\n\n## CLIPSegModel\n\n[[autodoc]] CLIPSegModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## CLIPSegTextModel\n\n[[autodoc]] CLIPSegTextModel\n    - forward\n\n## CLIPSegVisionModel\n\n[[autodoc]] CLIPSegVisionModel\n    - forward\n\n## CLIPSegForImageSegmentation\n\n[[autodoc]] CLIPSegForImageSegmentation\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1f786f8f-b835-412f-8881-ffe15a3f7d05
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/conditional_detr/", "title": "Conditional DETR", "tokens": 941, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1f786f8f-b835-412f-8881-ffe15a3f7d05\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/conditional_detr/\", \"title\": \"Conditional DETR\", \"tokens\": 941, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1af591f2-b793-4755-8225-49eae5539308\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/conditional_detr/\", \"title\": \"Conditional DETR\", \"tokens\": 941, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bf87211215e4be1a35d37849c53cfff899d0aef4b6a754c5f229aa24125a24b2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3451, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1af591f2-b793-4755-8225-49eae5539308", "doc_id": "1af591f2-b793-4755-8225-49eae5539308", "ref_doc_id": "1af591f2-b793-4755-8225-49eae5539308", "chroma:document": "# Conditional DETR\n\n## Overview\n\nThe Conditional DETR model was proposed in [Conditional DETR for Fast Training Convergence](https://arxiv.org/abs/2108.06152) by Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, Jingdong Wang. Conditional DETR presents a conditional cross-attention mechanism for fast DETR training. Conditional DETR converges 6.7\u00d7 to 10\u00d7 faster than DETR.\n\nThe abstract from the paper is the following:\n\n*The recently-developed DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings for localizing the four extremities and predicting the box, which increases the need for high-quality content embeddings and thus the training difficulty. Our approach, named conditional DETR, learns a conditional spatial query from the decoder embedding for decoder multi-head cross-attention. The benefit is that through the conditional spatial query, each cross-attention head is able to attend to a band containing a distinct region, e.g., one object extremity or a region inside the object box. This narrows down the spatial range for localizing the distinct regions for object classification and box regression, thus relaxing the dependence on the content embeddings and easing the training. Empirical results show that conditional DETR converges 6.7\u00d7 faster for the backbones R50 and R101 and 10\u00d7 faster for stronger backbones DC5-R50 and DC5-R101. Code is available at https://github.com/Atten4Vis/ConditionalDETR.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/conditional_detr_curve.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Conditional DETR shows much faster convergence compared to the original DETR. Taken from the <a href=\"https://arxiv.org/abs/2108.06152\">original paper</a>.</small>\n\nThis model was contributed by [DepuMeng](https://huggingface.co/DepuMeng). The original code can be found [here](https://github.com/Atten4Vis/ConditionalDETR).\n\n## Resources\n\n- Scripts for finetuning [`ConditionalDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\n## ConditionalDetrConfig\n\n[[autodoc]] ConditionalDetrConfig\n\n## ConditionalDetrImageProcessor\n\n[[autodoc]] ConditionalDetrImageProcessor\n    - preprocess\n    - post_process_object_detection\n    - post_process_instance_segmentation\n    - post_process_semantic_segmentation\n    - post_process_panoptic_segmentation\n\n## ConditionalDetrFeatureExtractor\n\n[[autodoc]] ConditionalDetrFeatureExtractor\n    - __call__\n    - post_process_object_detection\n    - post_process_instance_segmentation\n    - post_process_semantic_segmentation\n    - post_process_panoptic_segmentation\n\n## ConditionalDetrModel\n\n[[autodoc]] ConditionalDetrModel\n    - forward\n\n## ConditionalDetrForObjectDetection\n\n[[autodoc]] ConditionalDetrForObjectDetection\n    - forward\n\n## ConditionalDetrForSegmentation\n\n[[autodoc]] ConditionalDetrForSegmentation\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4f5ca3f3-98c7-4768-94c6-7108490cd746
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/visual_bert/", "title": "VisualBERT", "tokens": 1381, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4f5ca3f3-98c7-4768-94c6-7108490cd746\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert/\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6772e619-afe3-40cc-910a-decd3250cb00\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert/\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bd6a24b61b8714e396dd5f8147a37f2fac243351097979bbf37642e1b58390d1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d5877d34-6a5a-4cc1-8f57-7108ae4092fb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a09b812c6f15e320381f11029ec11514c916c322db8618c98a1de869df74b38b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3487, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6772e619-afe3-40cc-910a-decd3250cb00", "doc_id": "6772e619-afe3-40cc-910a-decd3250cb00", "ref_doc_id": "6772e619-afe3-40cc-910a-decd3250cb00", "chroma:document": "# VisualBERT\n\n## Overview\n\nThe VisualBERT model was proposed in [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://arxiv.org/pdf/1908.03557) by Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang.\nVisualBERT is a neural network trained on a variety of (image, text) pairs.\n\nThe abstract from the paper is the following:\n\n*We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks.\nVisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an\nassociated input image with self-attention. We further propose two visually-grounded language model objectives for\npre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2,\nand Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly\nsimpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any\nexplicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between\nverbs and image regions corresponding to their arguments.*\n\nThis model was contributed by [gchhablani](https://huggingface.co/gchhablani). The original code can be found [here](https://github.com/uclanlp/visualbert).\n\n## Usage tips\n\n1. Most of the checkpoints provided work with the [`VisualBertForPreTraining`] configuration. Other\n   checkpoints provided are the fine-tuned checkpoints for down-stream tasks - VQA ('visualbert-vqa'), VCR\n   ('visualbert-vcr'), NLVR2 ('visualbert-nlvr2'). Hence, if you are not working on these downstream tasks, it is\n   recommended that you use the pretrained checkpoints.\n\n2. For the VCR task, the authors use a fine-tuned detector for generating visual embeddings, for all the checkpoints.\n   We do not provide the detector and its weights as a part of the package, but it will be available in the research\n   projects, and the states can be loaded directly into the detector provided.\n\nVisualBERT is a multi-modal vision and language model. It can be used for visual question answering, multiple choice,\nvisual reasoning and region-to-phrase correspondence tasks. VisualBERT uses a BERT-like transformer to prepare\nembeddings for image-text pairs. Both the text and visual features are then projected to a latent space with identical\ndimension.\n\nTo feed images to the model, each image is passed through a pre-trained object detector and the regions and the\nbounding boxes are extracted. The authors use the features generated after passing these regions through a pre-trained\nCNN like ResNet as visual embeddings. They also add absolute position embeddings, and feed the resulting sequence of\nvectors to a standard BERT model. The text input is concatenated in the front of the visual embeddings in the embedding\nlayer, and is expected to be bound by [CLS] and a [SEP] tokens, as in BERT. The segment IDs must also be set\nappropriately for the textual and visual parts.\n\nThe [`BertTokenizer`] is used to encode the text. A custom detector/image processor must be used\nto get the visual embeddings. The following example notebooks show how to use VisualBERT with Detectron-like models:\n\n- [VisualBERT VQA demo notebook](https://github.com/huggingface/transformers/tree/main/examples/research_projects/visual_bert) : This notebook\n  contains an example on VisualBERT VQA."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d5877d34-6a5a-4cc1-8f57-7108ae4092fb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/visual_bert/", "title": "VisualBERT", "tokens": 1381, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d5877d34-6a5a-4cc1-8f57-7108ae4092fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert/\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6772e619-afe3-40cc-910a-decd3250cb00\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert/\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bd6a24b61b8714e396dd5f8147a37f2fac243351097979bbf37642e1b58390d1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4f5ca3f3-98c7-4768-94c6-7108490cd746\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert/\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"528453037f826a9fe864831f43bd418823085de086a334a7c5cd82ae48bd19c0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"72beaca6-ae18-45d5-9167-ccd4175ba7dc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"82ae1ddef447329ff73792fbbc2a5900dc8a64b2a29d3f35fbc88178c249e8c5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1668, \"end_char_idx\": 4092, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6772e619-afe3-40cc-910a-decd3250cb00", "doc_id": "6772e619-afe3-40cc-910a-decd3250cb00", "ref_doc_id": "6772e619-afe3-40cc-910a-decd3250cb00", "chroma:document": "Hence, if you are not working on these downstream tasks, it is\n   recommended that you use the pretrained checkpoints.\n\n2. For the VCR task, the authors use a fine-tuned detector for generating visual embeddings, for all the checkpoints.\n   We do not provide the detector and its weights as a part of the package, but it will be available in the research\n   projects, and the states can be loaded directly into the detector provided.\n\nVisualBERT is a multi-modal vision and language model. It can be used for visual question answering, multiple choice,\nvisual reasoning and region-to-phrase correspondence tasks. VisualBERT uses a BERT-like transformer to prepare\nembeddings for image-text pairs. Both the text and visual features are then projected to a latent space with identical\ndimension.\n\nTo feed images to the model, each image is passed through a pre-trained object detector and the regions and the\nbounding boxes are extracted. The authors use the features generated after passing these regions through a pre-trained\nCNN like ResNet as visual embeddings. They also add absolute position embeddings, and feed the resulting sequence of\nvectors to a standard BERT model. The text input is concatenated in the front of the visual embeddings in the embedding\nlayer, and is expected to be bound by [CLS] and a [SEP] tokens, as in BERT. The segment IDs must also be set\nappropriately for the textual and visual parts.\n\nThe [`BertTokenizer`] is used to encode the text. A custom detector/image processor must be used\nto get the visual embeddings. The following example notebooks show how to use VisualBERT with Detectron-like models:\n\n- [VisualBERT VQA demo notebook](https://github.com/huggingface/transformers/tree/main/examples/research_projects/visual_bert) : This notebook\n  contains an example on VisualBERT VQA.\n\n- [Generate Embeddings for VisualBERT (Colab Notebook)](https://colab.research.google.com/drive/1bLGxKdldwqnMVA5x4neY7-l_8fKGWQYI?usp=sharing) : This notebook contains\n  an example on how to generate visual embeddings.\n\nThe following example shows how to get the last hidden state using [`VisualBertModel`]:\n\n```python\n>>> import torch\n>>> from transformers import BertTokenizer, VisualBertModel\n\n>>> model = VisualBertModel.from_pretrained(\"uclanlp/visualbert-vqa-coco-pre\")\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n\n>>> inputs = tokenizer(\"What is the man eating?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 72beaca6-ae18-45d5-9167-ccd4175ba7dc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/visual_bert/", "title": "VisualBERT", "tokens": 1381, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"72beaca6-ae18-45d5-9167-ccd4175ba7dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert/\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6772e619-afe3-40cc-910a-decd3250cb00\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert/\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bd6a24b61b8714e396dd5f8147a37f2fac243351097979bbf37642e1b58390d1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d5877d34-6a5a-4cc1-8f57-7108ae4092fb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/visual_bert/\", \"title\": \"VisualBERT\", \"tokens\": 1381, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8f42e2b9ce02df5fcafaf6df93af759c923f074014cfbd6ef6b7701c43ddc0c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2463, \"end_char_idx\": 5287, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6772e619-afe3-40cc-910a-decd3250cb00", "doc_id": "6772e619-afe3-40cc-910a-decd3250cb00", "ref_doc_id": "6772e619-afe3-40cc-910a-decd3250cb00", "chroma:document": "To feed images to the model, each image is passed through a pre-trained object detector and the regions and the\nbounding boxes are extracted. The authors use the features generated after passing these regions through a pre-trained\nCNN like ResNet as visual embeddings. They also add absolute position embeddings, and feed the resulting sequence of\nvectors to a standard BERT model. The text input is concatenated in the front of the visual embeddings in the embedding\nlayer, and is expected to be bound by [CLS] and a [SEP] tokens, as in BERT. The segment IDs must also be set\nappropriately for the textual and visual parts.\n\nThe [`BertTokenizer`] is used to encode the text. A custom detector/image processor must be used\nto get the visual embeddings. The following example notebooks show how to use VisualBERT with Detectron-like models:\n\n- [VisualBERT VQA demo notebook](https://github.com/huggingface/transformers/tree/main/examples/research_projects/visual_bert) : This notebook\n  contains an example on VisualBERT VQA.\n\n- [Generate Embeddings for VisualBERT (Colab Notebook)](https://colab.research.google.com/drive/1bLGxKdldwqnMVA5x4neY7-l_8fKGWQYI?usp=sharing) : This notebook contains\n  an example on how to generate visual embeddings.\n\nThe following example shows how to get the last hidden state using [`VisualBertModel`]:\n\n```python\n>>> import torch\n>>> from transformers import BertTokenizer, VisualBertModel\n\n>>> model = VisualBertModel.from_pretrained(\"uclanlp/visualbert-vqa-coco-pre\")\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n\n>>> inputs = tokenizer(\"What is the man eating?\", return_tensors=\"pt\")\n>>> # this is a custom function that returns the visual embeddings given the image path\n>>> visual_embeds = get_visual_embeddings(image_path)\n\n>>> visual_token_type_ids = torch.ones(visual_embeds.shape[:-1], dtype=torch.long)\n>>> visual_attention_mask = torch.ones(visual_embeds.shape[:-1], dtype=torch.float)\n>>> inputs.update(\n...     {\n...         \"visual_embeds\": visual_embeds,\n...         \"visual_token_type_ids\": visual_token_type_ids,\n...         \"visual_attention_mask\": visual_attention_mask,\n...     }\n... )\n>>> outputs = model(**inputs)\n>>> last_hidden_state = outputs.last_hidden_state\n```\n\n## VisualBertConfig\n\n[[autodoc]] VisualBertConfig\n\n## VisualBertModel\n\n[[autodoc]] VisualBertModel\n    - forward\n\n## VisualBertForPreTraining\n\n[[autodoc]] VisualBertForPreTraining\n    - forward\n\n## VisualBertForQuestionAnswering\n\n[[autodoc]] VisualBertForQuestionAnswering\n    - forward\n\n## VisualBertForMultipleChoice\n\n[[autodoc]] VisualBertForMultipleChoice\n    - forward\n\n## VisualBertForVisualReasoning\n\n[[autodoc]] VisualBertForVisualReasoning\n    - forward\n\n## VisualBertForRegionToPhraseAlignment\n\n[[autodoc]] VisualBertForRegionToPhraseAlignment\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f4629a72-9d7d-4552-86d8-6ea6fd72f896
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus/", "title": "BigBirdPegasus", "tokens": 1076, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f4629a72-9d7d-4552-86d8-6ea6fd72f896\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus/\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c993acba-f034-44a5-854b-a6c0a0a66e47\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus/\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"992abe784a642245859da6763331e5b970a235566707b61367f197599aba41fc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"991f4fe1-5eb7-42a5-88e8-2bbae33f5acb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"353bd4fcf3a1e63f7c95dc6d74428844dbad4626a6eea493c5bd74fffef0656e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3099, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c993acba-f034-44a5-854b-a6c0a0a66e47", "doc_id": "c993acba-f034-44a5-854b-a6c0a0a66e47", "ref_doc_id": "c993acba-f034-44a5-854b-a6c0a0a66e47", "chroma:document": "# BigBirdPegasus\n\n## Overview\n\nThe BigBird model was proposed in [Big Bird: Transformers for Longer Sequences](https://arxiv.org/abs/2007.14062) by\nZaheer, Manzil and Guruganesh, Guru and Dubey, Kumar Avinava and Ainslie, Joshua and Alberti, Chris and Ontanon,\nSantiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and others. BigBird, is a sparse-attention\nbased transformer which extends Transformer based models, such as BERT to much longer sequences. In addition to sparse\nattention, BigBird also applies global attention as well as random attention to the input sequence. Theoretically, it\nhas been shown that applying sparse, global, and random attention approximates full attention, while being\ncomputationally much more efficient for longer sequences. As a consequence of the capability to handle longer context,\nBigBird has shown improved performance on various long document NLP tasks, such as question answering and\nsummarization, compared to BERT or RoBERTa.\n\nThe abstract from the paper is the following:\n\n*Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP.\nUnfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence\nlength due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that\nreduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and\nis Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our\ntheoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire\nsequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to\n8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context,\nBigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also\npropose novel applications to genomics data.*\n\nThe original code can be found [here](https://github.com/google-research/bigbird).\n\n## Usage tips\n\n- For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird).\n- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using\n  **original_full** is advised as there is no benefit in using **block_sparse** attention.\n- The code currently uses window size of 3 blocks and 2 global blocks.\n- Sequence length must be divisible by block size.\n- Current implementation supports only **ITC**.\n- Current implementation doesn't support **num_random_blocks = 0**.\n- BigBirdPegasus uses the [PegasusTokenizer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pegasus/tokenization_pegasus.py).\n- BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 991f4fe1-5eb7-42a5-88e8-2bbae33f5acb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus/", "title": "BigBirdPegasus", "tokens": 1076, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"991f4fe1-5eb7-42a5-88e8-2bbae33f5acb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus/\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c993acba-f034-44a5-854b-a6c0a0a66e47\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus/\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"992abe784a642245859da6763331e5b970a235566707b61367f197599aba41fc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f4629a72-9d7d-4552-86d8-6ea6fd72f896\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bigbird_pegasus/\", \"title\": \"BigBirdPegasus\", \"tokens\": 1076, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bca4e8dbaabbce3e2ea7bdaec6612ba5eb558dfec44e79591d0d44e939de740b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1312, \"end_char_idx\": 3960, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c993acba-f034-44a5-854b-a6c0a0a66e47", "doc_id": "c993acba-f034-44a5-854b-a6c0a0a66e47", "ref_doc_id": "c993acba-f034-44a5-854b-a6c0a0a66e47", "chroma:document": "To remedy this, we propose, BigBird, a sparse attention mechanism that\nreduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and\nis Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our\ntheoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire\nsequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to\n8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context,\nBigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also\npropose novel applications to genomics data.*\n\nThe original code can be found [here](https://github.com/google-research/bigbird).\n\n## Usage tips\n\n- For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird).\n- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using\n  **original_full** is advised as there is no benefit in using **block_sparse** attention.\n- The code currently uses window size of 3 blocks and 2 global blocks.\n- Sequence length must be divisible by block size.\n- Current implementation supports only **ITC**.\n- Current implementation doesn't support **num_random_blocks = 0**.\n- BigBirdPegasus uses the [PegasusTokenizer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pegasus/tokenization_pegasus.py).\n- BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## BigBirdPegasusConfig\n\n[[autodoc]] BigBirdPegasusConfig\n    - all\n\n## BigBirdPegasusModel\n\n[[autodoc]] BigBirdPegasusModel\n    - forward\n\n## BigBirdPegasusForConditionalGeneration\n\n[[autodoc]] BigBirdPegasusForConditionalGeneration\n    - forward\n\n## BigBirdPegasusForSequenceClassification\n\n[[autodoc]] BigBirdPegasusForSequenceClassification\n    - forward\n\n## BigBirdPegasusForQuestionAnswering\n\n[[autodoc]] BigBirdPegasusForQuestionAnswering\n    - forward\n\n## BigBirdPegasusForCausalLM\n\n[[autodoc]] BigBirdPegasusForCausalLM\n    - forward"}


