# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 195f809f-54d7-4806-a1b7-afd9aa8b7919
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapex", "title": "TAPEX", "tokens": 1845, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"195f809f-54d7-4806-a1b7-afd9aa8b7919\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87ed64d5-7c96-4532-8692-0b8ae525ed33\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"89187f4979b4b5c48d93d82c138bbaf27bc22127d5885139bc7536f6f82c168c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f152c570-0557-4387-9b78-c0a00024dd9d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapex\", \"title\": \"TAPEX\", \"tokens\": 1845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"39b57cbe883d79149dc3170362cb5ae15f525e7dcfe1f25ce2301bc5eb98401b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4159, \"end_char_idx\": 6924, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "doc_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "ref_doc_id": "87ed64d5-7c96-4532-8692-0b8ae525ed33", "chroma:document": ">>> encoding = tokenizer(table, question, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> predicted_answer = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]\n>>> print(predicted_answer)\n53\n```\n\nNote that [`TapexTokenizer`] also supports batched inference. Hence, one can provide a batch of different tables/questions, or a batch of a single table\nand multiple questions, or a batch of a single query and multiple tables. Let's illustrate this:\n\n```python\n>>> # prepare table + question\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> questions = [\n...     \"how many movies does Leonardo Di Caprio have?\",\n...     \"which actor has 69 movies?\",\n...     \"what's the first name of the actor who has 87 movies?\",\n... ]\n>>> encoding = tokenizer(table, questions, padding=True, return_tensors=\"pt\")\n\n>>> # let the model generate an answer autoregressively\n>>> outputs = model.generate(**encoding)\n\n>>> # decode back to text\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n[' 53', ' george clooney', ' brad pitt']\n```\n\nIn case one wants to do table verification (i.e. the task of determining whether a given sentence is supported or refuted by the contents\nof a table), one can instantiate a [`BartForSequenceClassification`] model. TAPEX has checkpoints on the hub fine-tuned on TabFact, an important\nbenchmark for table fact checking (it achieves 84% accuracy). The code example below again leverages the [Auto API](auto).\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/tapex-large-finetuned-tabfact\")\n>>> model = AutoModelForSequenceClassification.from_pretrained(\"microsoft/tapex-large-finetuned-tabfact\")\n\n>>> # prepare table + sentence\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> table = pd.DataFrame.from_dict(data)\n>>> sentence = \"George Clooney has 30 movies\"\n\n>>> encoding = tokenizer(table, sentence, return_tensors=\"pt\")\n\n>>> # forward pass\n>>> outputs = model(**encoding)\n\n>>> # print prediction\n>>> predicted_class_idx = outputs.logits[0].argmax(dim=0).item()\n>>> print(model.config.id2label[predicted_class_idx])\nRefused\n```\n\n<Tip> \n\nTAPEX architecture is the same as BART, except for tokenization. Refer to [BART documentation](bart) for information on \nconfiguration classes and their parameters. TAPEX-specific tokenizer is documented below.  \n\n</Tip>\n\n## TapexTokenizer\n\n[[autodoc]] TapexTokenizer\n    - __call__\n    - save_vocabulary"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 86a43461-14a6-4268-a403-7bd6191f5ce5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/efficientformer", "title": "EfficientFormer", "tokens": 1010, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"86a43461-14a6-4268-a403-7bd6191f5ce5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba685923-7998-43ae-ad23-4ce9b485e9e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3cd81155567b54b56291efa2b4d62404dbe65d0e7d9fe8339525aec4ec6e7928\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a7afa18a-8caa-44d2-b05e-97bae416e6f7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ec14cfc786519fc9b478c6c1180b2a63cfb0a34f01a147f1a522728e6af675e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2811, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba685923-7998-43ae-ad23-4ce9b485e9e1", "doc_id": "ba685923-7998-43ae-ad23-4ce9b485e9e1", "ref_doc_id": "ba685923-7998-43ae-ad23-4ce9b485e9e1", "chroma:document": "# EfficientFormer\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe EfficientFormer model was proposed in [EfficientFormer: Vision Transformers at MobileNet Speed](https://arxiv.org/abs/2206.01191)\nby Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, Jian Ren.  EfficientFormer proposes a\ndimension-consistent pure transformer that can be run on mobile devices for dense prediction tasks like image classification, object\ndetection and semantic segmentation.\n\nThe abstract from the paper is the following:\n\n*Vision Transformers (ViT) have shown rapid progress in computer vision tasks, achieving promising results on various benchmarks.\nHowever, due to the massive number of parameters and model design, e.g., attention mechanism, ViT-based models are generally\ntimes slower than lightweight convolutional networks. Therefore, the deployment of ViT for real-time applications is particularly\nchallenging, especially on resource-constrained hardware such as mobile devices. Recent efforts try to reduce the computation\ncomplexity of ViT through network architecture search or hybrid design with MobileNet block, yet the inference speed is still\nunsatisfactory. This leads to an important question: can transformers run as fast as MobileNet while obtaining high performance?\nTo answer this, we first revisit the network architecture and operators used in ViT-based models and identify inefficient designs.\nThen we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm.\nFinally, we perform latency-driven slimming to get a series of final models dubbed EfficientFormer.\nExtensive experiments show the superiority of EfficientFormer in performance and speed on mobile devices.\nOur fastest model, EfficientFormer-L1, achieves 79.2% top-1 accuracy on ImageNet-1K with only 1.6 ms inference latency on\niPhone 12 (compiled with CoreML), which { runs as fast as MobileNetV2\u00d71.4 (1.6 ms, 74.7% top-1),} and our largest model,\nEfficientFormer-L7, obtains 83.3% accuracy with only 7.0 ms latency. Our work proves that properly designed transformers can\nreach extremely low latency on mobile devices while maintaining high performance.*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03) and [Bearnardd](https://huggingface.co/Bearnardd).\nThe original code can be found [here](https://github.com/snap-research/EfficientFormer). The TensorFlow version of this model was added by [D-Roberts](https://huggingface.co/D-Roberts)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a7afa18a-8caa-44d2-b05e-97bae416e6f7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/efficientformer", "title": "EfficientFormer", "tokens": 1010, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a7afa18a-8caa-44d2-b05e-97bae416e6f7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba685923-7998-43ae-ad23-4ce9b485e9e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3cd81155567b54b56291efa2b4d62404dbe65d0e7d9fe8339525aec4ec6e7928\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"86a43461-14a6-4268-a403-7bd6191f5ce5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/efficientformer\", \"title\": \"EfficientFormer\", \"tokens\": 1010, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"128ff2b7e5edfb2d24609210290837135328c4192b23c9e2920c7f11a2847b4d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1148, \"end_char_idx\": 3747, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba685923-7998-43ae-ad23-4ce9b485e9e1", "doc_id": "ba685923-7998-43ae-ad23-4ce9b485e9e1", "ref_doc_id": "ba685923-7998-43ae-ad23-4ce9b485e9e1", "chroma:document": "Therefore, the deployment of ViT for real-time applications is particularly\nchallenging, especially on resource-constrained hardware such as mobile devices. Recent efforts try to reduce the computation\ncomplexity of ViT through network architecture search or hybrid design with MobileNet block, yet the inference speed is still\nunsatisfactory. This leads to an important question: can transformers run as fast as MobileNet while obtaining high performance?\nTo answer this, we first revisit the network architecture and operators used in ViT-based models and identify inefficient designs.\nThen we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm.\nFinally, we perform latency-driven slimming to get a series of final models dubbed EfficientFormer.\nExtensive experiments show the superiority of EfficientFormer in performance and speed on mobile devices.\nOur fastest model, EfficientFormer-L1, achieves 79.2% top-1 accuracy on ImageNet-1K with only 1.6 ms inference latency on\niPhone 12 (compiled with CoreML), which { runs as fast as MobileNetV2\u00d71.4 (1.6 ms, 74.7% top-1),} and our largest model,\nEfficientFormer-L7, obtains 83.3% accuracy with only 7.0 ms latency. Our work proves that properly designed transformers can\nreach extremely low latency on mobile devices while maintaining high performance.*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03) and [Bearnardd](https://huggingface.co/Bearnardd).\nThe original code can be found [here](https://github.com/snap-research/EfficientFormer). The TensorFlow version of this model was added by [D-Roberts](https://huggingface.co/D-Roberts).\n\n## Documentation resources\n\n- [Image classification task guide](../tasks/image_classification)\n\n## EfficientFormerConfig\n\n[[autodoc]] EfficientFormerConfig\n\n## EfficientFormerImageProcessor\n\n[[autodoc]] EfficientFormerImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## EfficientFormerModel\n\n[[autodoc]] EfficientFormerModel\n    - forward\n\n## EfficientFormerForImageClassification\n\n[[autodoc]] EfficientFormerForImageClassification\n    - forward\n\n## EfficientFormerForImageClassificationWithTeacher\n\n[[autodoc]] EfficientFormerForImageClassificationWithTeacher\n    - forward\n\n</pt>\n<tf>\n\n## TFEfficientFormerModel\n\n[[autodoc]] TFEfficientFormerModel\n    - call\n\n## TFEfficientFormerForImageClassification\n\n[[autodoc]] TFEfficientFormerForImageClassification\n    - call\n\n## TFEfficientFormerForImageClassificationWithTeacher\n\n[[autodoc]] TFEfficientFormerForImageClassificationWithTeacher\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 66346b56-0cec-4d4f-8395-4c53b1dc12f7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/madlad-400", "title": "MADLAD-400", "tokens": 797, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"66346b56-0cec-4d4f-8395-4c53b1dc12f7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/madlad-400\", \"title\": \"MADLAD-400\", \"tokens\": 797, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf46428a-71e3-4a29-87c9-d7d7b56f3be6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/madlad-400\", \"title\": \"MADLAD-400\", \"tokens\": 797, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"928ac9205122fd6550dd8ba6e5178fd7963b56d0421266d01685adb376c5f2cb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2495, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf46428a-71e3-4a29-87c9-d7d7b56f3be6", "doc_id": "bf46428a-71e3-4a29-87c9-d7d7b56f3be6", "ref_doc_id": "bf46428a-71e3-4a29-87c9-d7d7b56f3be6", "chroma:document": "# MADLAD-400\n\n## Overview\n\nMADLAD-400 models were released in the paper [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](MADLAD-400: A Multilingual And Document-Level Large Audited Dataset). \n\nThe abstract from the paper is the following: \n\n*We introduce MADLAD-400, a manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss \nthe limitations revealed by self-auditing MADLAD-400, and the role data auditing\nhad in the dataset creation process. We then train and release a 10.7B-parameter\nmultilingual machine translation model on 250 billion tokens covering over 450\nlanguages using publicly available data, and find that it is competitive with models\nthat are significantly larger, and report the results on different domains. In addition, we train a 8B-parameter language model, and assess the results on few-shot\ntranslation. We make the baseline models 1\navailable to the research community.*\n\nThis model was added by [Juarez Bochi](https://huggingface.co/jbochi). The original checkpoints can be found [here](https://github.com/google-research/google-research/tree/master/madlad_400). \n\nThis is a machine translation model that supports many low-resource languages, and that is competitive with models that are significantly larger.\n\nOne can directly use MADLAD-400 weights without finetuning the model:\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/madlad400-3b-mt\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/madlad400-3b-mt\")\n\n>>> inputs = tokenizer(\"<2pt> I love pizza!\", return_tensors=\"pt\")\n>>> outputs = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['Eu amo pizza!']\n```\n\nGoogle has released the following variants:\n\n- [google/madlad400-3b-mt](https://huggingface.co/google/madlad400-3b-mt)\n\n- [google/madlad400-7b-mt](https://huggingface.co/google/madlad400-7b-mt)\n\n- [google/madlad400-7b-mt-bt](https://huggingface.co/google/madlad400-7b-mt-bt)\n\n- [google/madlad400-10b-mt](https://huggingface.co/google/madlad400-10b-mt)\n\nThe original checkpoints can be found [here](https://github.com/google-research/google-research/tree/master/madlad_400).\n\n<Tip>\n\nRefer to [T5's documentation page](t5) for all API references, code examples, and notebooks. For more details regarding training and evaluation of the MADLAD-400, refer to the model card.\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cb9c8972-ec53-419d-a72f-a9079499dfd4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mamba", "title": "Mamba", "tokens": 1262, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cb9c8972-ec53-419d-a72f-a9079499dfd4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c5e6091-ec66-4c18-a7ed-4cf82ee7a5a9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36b88bcfc8d36c0893f80052c680ef378e63f05dc76b058b9b0cd90d61079e68\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6659cb29-dbbd-4fe3-9c14-28bcbdc9501d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"95b10197a6a934ae901fe83bf7f516c171a5e8b9656efa9e72d8b2843af7c9f4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3508, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c5e6091-ec66-4c18-a7ed-4cf82ee7a5a9", "doc_id": "0c5e6091-ec66-4c18-a7ed-4cf82ee7a5a9", "ref_doc_id": "0c5e6091-ec66-4c18-a7ed-4cf82ee7a5a9", "chroma:document": "# Mamba\n\n## Overview\n\nThe Mamba model was proposed in [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752) by Albert Gu and Tri Dao.\n\nThis model is a new paradigm architecture based on `state-space-models`. You can read more about the intuition behind these [here](https://srush.github.io/annotated-s4/).\n\nThe abstract from the paper is the following:\n\n*Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5\u00d7 higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.*\n\nTips:\n\n- Mamba is a new `state space model` architecture that rivals the classic Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of [FlashAttention](https://github.com/Dao-AILab/flash-attention).\n- Mamba stacks `mixer` layers, which are the equivalent of `Attention` layers. The core logic of `mamba` is held in the `MambaMixer` class.\n- Two implementations cohabit: one is optimized and uses fast cuda kernels, while the other one is naive but can run on any device!\n- The current implementation leverages the original cuda kernels: the equivalent of flash attention for Mamba are hosted in the [`mamba-ssm`](https://github.com/state-spaces/mamba) and the [`causal_conv1d`](https://github.com/Dao-AILab/causal-conv1d) repositories. Make sure to install them if your hardware supports them!\n- Contributions to make the naive path faster are welcome \ud83e\udd17\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/state-spaces/mamba).\n\n# Usage\n\n### A simple generation example: \n```python \nfrom transformers import MambaConfig, MambaForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"state-spaces/mamba-130m-hf\")\nmodel = MambaForCausalLM.from_pretrained(\"state-spaces/mamba-130m-hf\")\ninput_ids = tokenizer(\"Hey how are you doing?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6659cb29-dbbd-4fe3-9c14-28bcbdc9501d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mamba", "title": "Mamba", "tokens": 1262, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6659cb29-dbbd-4fe3-9c14-28bcbdc9501d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c5e6091-ec66-4c18-a7ed-4cf82ee7a5a9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36b88bcfc8d36c0893f80052c680ef378e63f05dc76b058b9b0cd90d61079e68\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb9c8972-ec53-419d-a72f-a9079499dfd4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba\", \"title\": \"Mamba\", \"tokens\": 1262, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3b281fee2e8844562e46aaaa5de5bdaed408c0c2b6b91298a4a2810f147829bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1896, \"end_char_idx\": 4850, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c5e6091-ec66-4c18-a7ed-4cf82ee7a5a9", "doc_id": "0c5e6091-ec66-4c18-a7ed-4cf82ee7a5a9", "ref_doc_id": "0c5e6091-ec66-4c18-a7ed-4cf82ee7a5a9", "chroma:document": "On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.*\n\nTips:\n\n- Mamba is a new `state space model` architecture that rivals the classic Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of [FlashAttention](https://github.com/Dao-AILab/flash-attention).\n- Mamba stacks `mixer` layers, which are the equivalent of `Attention` layers. The core logic of `mamba` is held in the `MambaMixer` class.\n- Two implementations cohabit: one is optimized and uses fast cuda kernels, while the other one is naive but can run on any device!\n- The current implementation leverages the original cuda kernels: the equivalent of flash attention for Mamba are hosted in the [`mamba-ssm`](https://github.com/state-spaces/mamba) and the [`causal_conv1d`](https://github.com/Dao-AILab/causal-conv1d) repositories. Make sure to install them if your hardware supports them!\n- Contributions to make the naive path faster are welcome \ud83e\udd17\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/state-spaces/mamba).\n\n# Usage\n\n### A simple generation example: \n```python \nfrom transformers import MambaConfig, MambaForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"state-spaces/mamba-130m-hf\")\nmodel = MambaForCausalLM.from_pretrained(\"state-spaces/mamba-130m-hf\")\ninput_ids = tokenizer(\"Hey how are you doing?\", return_tensors= \"pt\")[\"input_ids\"]\n\nout = model.generate(input_ids, max_new_tokens=10)\nprint(tokenizer.batch_decode(out))\n```\n\n### Peft finetuning\nThe slow version is not very stable for training, and the fast one needs `float32`!\n\n```python \nfrom datasets import load_dataset\nfrom trl import SFTTrainer\nfrom peft import LoraConfig\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments\nmodel_id = \"state-spaces/mamba-130m-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\ndataset = load_dataset(\"Abirate/english_quotes\", split=\"train\")\ntraining_args = TrainingArguments(\n    output_dir=\"./results\",\n    num_train_epochs=3,\n    per_device_train_batch_size=4,\n    logging_dir='./logs',\n    logging_steps=10,\n    learning_rate=2e-3\n)\nlora_config =  LoraConfig(\n        r=8,\n        target_modules=[\"x_proj\", \"embeddings\", \"in_proj\", \"out_proj\"],\n        task_type=\"CAUSAL_LM\",\n        bias=\"none\"\n)\ntrainer = SFTTrainer(\n    model=model,\n    tokenizer=tokenizer,\n    args=training_args,\n    peft_config=lora_config,\n    train_dataset=dataset,\n    dataset_text_field=\"quote\",\n)\ntrainer.train()\n```\n\n## MambaConfig\n\n[[autodoc]] MambaConfig\n\n## MambaModel\n\n[[autodoc]] MambaModel\n    - forward\n\n## MambaLMHeadModel\n\n[[autodoc]] MambaForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7958f8f1-a8cb-4dd8-abbe-8ad8b44aeece
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cvt", "title": "Convolutional Vision Transformer (CvT)", "tokens": 1132, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7958f8f1-a8cb-4dd8-abbe-8ad8b44aeece\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a3335e2c-3ca8-4fbd-9794-d26ca00aca79\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"19b75c55e1f3611dfb7fee79f3a3c4807b8513c46156629f3e15657b28317bc2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25579981-d418-408e-ab87-babe015c9aee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"74216f69059be64d5a69b28258bc72bb5fb25be6b43e90b714a897fcc53a4415\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3034, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a3335e2c-3ca8-4fbd-9794-d26ca00aca79", "doc_id": "a3335e2c-3ca8-4fbd-9794-d26ca00aca79", "ref_doc_id": "a3335e2c-3ca8-4fbd-9794-d26ca00aca79", "chroma:document": "# Convolutional Vision Transformer (CvT)\n\n## Overview\n\nThe CvT model was proposed in [CvT: Introducing Convolutions to Vision Transformers](https://arxiv.org/abs/2103.15808) by Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan and Lei Zhang. The Convolutional vision Transformer (CvT) improves the [Vision Transformer (ViT)](vit) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs.\n\nThe abstract from the paper is the following:\n\n*We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) \nin performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through \ntwo primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer \nblock leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) \nto the ViT architecture (\\ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (\\ie dynamic attention, \nglobal context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves \nstate-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, \nperformance gains are maintained when pretrained on larger datasets (\\eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on \nImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, \na crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks.*\n\nThis model was contributed by [anugunj](https://huggingface.co/anugunj). The original code can be found [here](https://github.com/microsoft/CvT).\n\n## Usage tips\n\n- CvT models are regular Vision Transformers, but trained with convolutions. They outperform the [original model (ViT)](vit) when fine-tuned on ImageNet-1K and CIFAR-100.\n- You can check out demo notebooks regarding inference as well as fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace [`ViTFeatureExtractor`] by [`AutoImageProcessor`] and [`ViTForImageClassification`] by [`CvtForImageClassification`]).\n- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of 14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CvT."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 25579981-d418-408e-ab87-babe015c9aee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cvt", "title": "Convolutional Vision Transformer (CvT)", "tokens": 1132, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"25579981-d418-408e-ab87-babe015c9aee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a3335e2c-3ca8-4fbd-9794-d26ca00aca79\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"19b75c55e1f3611dfb7fee79f3a3c4807b8513c46156629f3e15657b28317bc2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7958f8f1-a8cb-4dd8-abbe-8ad8b44aeece\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cvt\", \"title\": \"Convolutional Vision Transformer (CvT)\", \"tokens\": 1132, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"22ef80bd85f7e185da7cffaa3b3e258a7e80527cc9e97f3115e74a91e82cdfac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1569, \"end_char_idx\": 4044, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a3335e2c-3ca8-4fbd-9794-d26ca00aca79", "doc_id": "a3335e2c-3ca8-4fbd-9794-d26ca00aca79", "ref_doc_id": "a3335e2c-3ca8-4fbd-9794-d26ca00aca79", "chroma:document": "Pre-trained on \nImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, \na crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks.*\n\nThis model was contributed by [anugunj](https://huggingface.co/anugunj). The original code can be found [here](https://github.com/microsoft/CvT).\n\n## Usage tips\n\n- CvT models are regular Vision Transformers, but trained with convolutions. They outperform the [original model (ViT)](vit) when fine-tuned on ImageNet-1K and CIFAR-100.\n- You can check out demo notebooks regarding inference as well as fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace [`ViTFeatureExtractor`] by [`AutoImageProcessor`] and [`ViTForImageClassification`] by [`CvtForImageClassification`]).\n- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of 14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CvT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`CvtForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## CvtConfig\n\n[[autodoc]] CvtConfig\n\n<frameworkcontent>\n<pt>\n\n## CvtModel\n\n[[autodoc]] CvtModel\n    - forward\n\n## CvtForImageClassification\n\n[[autodoc]] CvtForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFCvtModel\n\n[[autodoc]] TFCvtModel\n    - call\n\n## TFCvtForImageClassification\n\n[[autodoc]] TFCvtForImageClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ea4b3fba-2b43-47fe-907e-e75424cb3008
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinov2", "title": "DINOv2", "tokens": 1250, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ea4b3fba-2b43-47fe-907e-e75424cb3008\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2\", \"title\": \"DINOv2\", \"tokens\": 1250, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"808ed18e-7864-4a33-a657-bc15b99b4fae\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2\", \"title\": \"DINOv2\", \"tokens\": 1250, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff46a2c1889ffe4c0989b3292e7b1a5cc5952913becf41a895780e377576c78d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"290bc3a3-be98-455d-92ad-ff4764d29cde\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ded34696c8b883b766e71796f6893c57bd9f4f451b03d9541883ac8fc42254c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2517, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "808ed18e-7864-4a33-a657-bc15b99b4fae", "doc_id": "808ed18e-7864-4a33-a657-bc15b99b4fae", "ref_doc_id": "808ed18e-7864-4a33-a657-bc15b99b4fae", "chroma:document": "# DINOv2\n\n## Overview\n\nThe DINOv2 model was proposed in [DINOv2: Learning Robust Visual Features without Supervision](https://arxiv.org/abs/2304.07193) by\nMaxime Oquab, Timoth\u00e9e Darcet, Th\u00e9o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herv\u00e9 Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, Piotr Bojanowski.\nDINOv2 is an upgrade of [DINO](https://arxiv.org/abs/2104.14294), a self-supervised method applied on [Vision Transformers](vit). This method enables all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning.\n\nThe abstract from the paper is the following:\n\n*The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/facebookresearch/dinov2).\n\n## Usage tips\n\nThe model can be traced using `torch.jit.trace` which leverages JIT compilation to optimize the model making it faster to run. Note this still produces some mis-matched elements and the difference between the original model and the traced model is of the order of 1e-4."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 290bc3a3-be98-455d-92ad-ff4764d29cde
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinov2", "title": "DINOv2", "tokens": 1250, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"290bc3a3-be98-455d-92ad-ff4764d29cde\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2\", \"title\": \"DINOv2\", \"tokens\": 1250, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"808ed18e-7864-4a33-a657-bc15b99b4fae\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2\", \"title\": \"DINOv2\", \"tokens\": 1250, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff46a2c1889ffe4c0989b3292e7b1a5cc5952913becf41a895780e377576c78d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ea4b3fba-2b43-47fe-907e-e75424cb3008\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2\", \"title\": \"DINOv2\", \"tokens\": 1250, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"52f6b9213a01a46ddf4146cf9201f946e314cf9aad500fe6642d06bbfa57e4df\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"01c9acc0-ce37-4913-9a5c-47c57603d954\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9dd13c5431a9f7ddc8ea81659fef745d588a61a9a99dcbecf54c519259c13806\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 681, \"end_char_idx\": 4183, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "808ed18e-7864-4a33-a657-bc15b99b4fae", "doc_id": "808ed18e-7864-4a33-a657-bc15b99b4fae", "ref_doc_id": "808ed18e-7864-4a33-a657-bc15b99b4fae", "chroma:document": "This method enables all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning.\n\nThe abstract from the paper is the following:\n\n*The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/facebookresearch/dinov2).\n\n## Usage tips\n\nThe model can be traced using `torch.jit.trace` which leverages JIT compilation to optimize the model making it faster to run. Note this still produces some mis-matched elements and the difference between the original model and the traced model is of the order of 1e-4.\n\n```python\nimport torch\nfrom transformers import AutoImageProcessor, AutoModel\nfrom PIL import Image\nimport requests\n\nurl = 'http://images.cocodataset.org/val2017/000000039769.jpg'\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprocessor = AutoImageProcessor.from_pretrained('facebook/dinov2-base')\nmodel = AutoModel.from_pretrained('facebook/dinov2-base')\n\ninputs = processor(images=image, return_tensors=\"pt\")\noutputs = model(**inputs)\nlast_hidden_states = outputs[0]\n\n# We have to force return_dict=False for tracing\nmodel.config.return_dict = False\n\nwith torch.no_grad():\n    traced_model = torch.jit.trace(model, [inputs.pixel_values])\n    traced_outputs = traced_model(inputs.pixel_values)\n\nprint((last_hidden_states - traced_outputs[0]).abs().max())\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DINOv2.\n\n- Demo notebooks for DINOv2 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DINOv2). \ud83c\udf0e\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`Dinov2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 01c9acc0-ce37-4913-9a5c-47c57603d954
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dinov2", "title": "DINOv2", "tokens": 1250, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"01c9acc0-ce37-4913-9a5c-47c57603d954\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2\", \"title\": \"DINOv2\", \"tokens\": 1250, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"808ed18e-7864-4a33-a657-bc15b99b4fae\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2\", \"title\": \"DINOv2\", \"tokens\": 1250, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff46a2c1889ffe4c0989b3292e7b1a5cc5952913becf41a895780e377576c78d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"290bc3a3-be98-455d-92ad-ff4764d29cde\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dinov2\", \"title\": \"DINOv2\", \"tokens\": 1250, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61ab7a4665c9d85a7d2b8697d0ceb1cc34bc164e281b9c26222dba57d8034acf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2519, \"end_char_idx\": 4599, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "808ed18e-7864-4a33-a657-bc15b99b4fae", "doc_id": "808ed18e-7864-4a33-a657-bc15b99b4fae", "ref_doc_id": "808ed18e-7864-4a33-a657-bc15b99b4fae", "chroma:document": "```python\nimport torch\nfrom transformers import AutoImageProcessor, AutoModel\nfrom PIL import Image\nimport requests\n\nurl = 'http://images.cocodataset.org/val2017/000000039769.jpg'\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprocessor = AutoImageProcessor.from_pretrained('facebook/dinov2-base')\nmodel = AutoModel.from_pretrained('facebook/dinov2-base')\n\ninputs = processor(images=image, return_tensors=\"pt\")\noutputs = model(**inputs)\nlast_hidden_states = outputs[0]\n\n# We have to force return_dict=False for tracing\nmodel.config.return_dict = False\n\nwith torch.no_grad():\n    traced_model = torch.jit.trace(model, [inputs.pixel_values])\n    traced_outputs = traced_model(inputs.pixel_values)\n\nprint((last_hidden_states - traced_outputs[0]).abs().max())\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DINOv2.\n\n- Demo notebooks for DINOv2 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DINOv2). \ud83c\udf0e\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`Dinov2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## Dinov2Config\n\n[[autodoc]] Dinov2Config\n\n<frameworkcontent>\n<pt>\n\n## Dinov2Model\n\n[[autodoc]] Dinov2Model\n    - forward\n\n## Dinov2ForImageClassification\n\n[[autodoc]] Dinov2ForImageClassification\n    - forward\n\n</pt>\n<jax>\n\n## FlaxDinov2Model\n\n[[autodoc]] FlaxDinov2Model\n    - __call__\n\n\n## FlaxDinov2ForImageClassification\n\n[[autodoc]] FlaxDinov2ForImageClassification\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 377476f3-ccf1-4f42-8cf5-5ff0078d0137
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/univnet", "title": "UnivNet", "tokens": 1204, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"377476f3-ccf1-4f42-8cf5-5ff0078d0137\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"29079440-7ecd-40cd-8ca0-f97f12f06640\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3698223795854e2364f7e4fa0e3647e6067aff9bd4a8ae3fec72690ca5ce5cd2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"85c83b19-5378-4efd-b2de-2282d346bda4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"643cc3abefefb7f891bdb15990ae11608f90361a279e08465d2c63081fbc079e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3045, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "29079440-7ecd-40cd-8ca0-f97f12f06640", "doc_id": "29079440-7ecd-40cd-8ca0-f97f12f06640", "ref_doc_id": "29079440-7ecd-40cd-8ca0-f97f12f06640", "chroma:document": "# UnivNet\n\n## Overview\n\nThe UnivNet model was proposed in [UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation](https://arxiv.org/abs/2106.07889) by Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kin, and Juntae Kim.\nThe UnivNet model is a generative adversarial network (GAN) trained to synthesize high fidelity speech waveforms. The UnivNet model shared in `transformers` is the *generator*, which maps a conditioning log-mel spectrogram and optional noise sequence to a speech waveform (e.g. a vocoder). Only the generator is required for inference. The *discriminator* used to train the `generator` is not implemented.\n\nThe abstract from the paper is the following:\n\n*Most neural vocoders employ band-limited mel-spectrograms to generate waveforms. If full-band spectral features are used as the input, the vocoder can be provided with as much acoustic information as possible. However, in some models employing full-band mel-spectrograms, an over-smoothing problem occurs as part of which non-sharp spectrograms are generated. To address this problem, we propose UnivNet, a neural vocoder that synthesizes high-fidelity waveforms in real time. Inspired by works in the field of voice activity detection, we added a multi-resolution spectrogram discriminator that employs multiple linear spectrogram magnitudes computed using various parameter sets. Using full-band mel-spectrograms as input, we expect to generate high-resolution signals by adding a discriminator that employs spectrograms of multiple resolutions as the input. In an evaluation on a dataset containing information on hundreds of speakers, UnivNet obtained the best objective and subjective results among competing models for both seen and unseen speakers. These results, including the best subjective score for text-to-speech, demonstrate the potential for fast adaptation to new speakers without a need for training from scratch.*\n\nTips:\n\n- The `noise_sequence` argument for [`UnivNetModel.forward`] should be standard Gaussian noise (such as from `torch.randn`) of shape `([batch_size], noise_length, model.config.model_in_channels)`, where `noise_length` should match the length dimension (dimension 1) of the `input_features` argument. If not supplied, it will be randomly generated; a `torch.Generator` can be supplied to the `generator` argument so that the forward pass can be reproduced. (Note that [`UnivNetFeatureExtractor`] will return generated noise by default, so it shouldn't be necessary to generate `noise_sequence` manually.)\n- Padding added by [`UnivNetFeatureExtractor`] can be removed from the [`UnivNetModel`] output through the [`UnivNetFeatureExtractor.batch_decode`] method, as shown in the usage example below.\n- Padding the end of each waveform with silence can reduce artifacts at the end of the generated audio sample. This can be done by supplying `pad_end = True` to [`UnivNetFeatureExtractor.__call__`]. See [this issue](https://github.com/seungwonpark/melgan/issues/8) for more details."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 85c83b19-5378-4efd-b2de-2282d346bda4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/univnet", "title": "UnivNet", "tokens": 1204, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"85c83b19-5378-4efd-b2de-2282d346bda4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"29079440-7ecd-40cd-8ca0-f97f12f06640\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3698223795854e2364f7e4fa0e3647e6067aff9bd4a8ae3fec72690ca5ce5cd2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"377476f3-ccf1-4f42-8cf5-5ff0078d0137\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/univnet\", \"title\": \"UnivNet\", \"tokens\": 1204, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ac0a98ed7b6ee877e4cfa251949dda81e73e6a01f20ee820df793f40348b15e3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1203, \"end_char_idx\": 4646, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "29079440-7ecd-40cd-8ca0-f97f12f06640", "doc_id": "29079440-7ecd-40cd-8ca0-f97f12f06640", "ref_doc_id": "29079440-7ecd-40cd-8ca0-f97f12f06640", "chroma:document": "Inspired by works in the field of voice activity detection, we added a multi-resolution spectrogram discriminator that employs multiple linear spectrogram magnitudes computed using various parameter sets. Using full-band mel-spectrograms as input, we expect to generate high-resolution signals by adding a discriminator that employs spectrograms of multiple resolutions as the input. In an evaluation on a dataset containing information on hundreds of speakers, UnivNet obtained the best objective and subjective results among competing models for both seen and unseen speakers. These results, including the best subjective score for text-to-speech, demonstrate the potential for fast adaptation to new speakers without a need for training from scratch.*\n\nTips:\n\n- The `noise_sequence` argument for [`UnivNetModel.forward`] should be standard Gaussian noise (such as from `torch.randn`) of shape `([batch_size], noise_length, model.config.model_in_channels)`, where `noise_length` should match the length dimension (dimension 1) of the `input_features` argument. If not supplied, it will be randomly generated; a `torch.Generator` can be supplied to the `generator` argument so that the forward pass can be reproduced. (Note that [`UnivNetFeatureExtractor`] will return generated noise by default, so it shouldn't be necessary to generate `noise_sequence` manually.)\n- Padding added by [`UnivNetFeatureExtractor`] can be removed from the [`UnivNetModel`] output through the [`UnivNetFeatureExtractor.batch_decode`] method, as shown in the usage example below.\n- Padding the end of each waveform with silence can reduce artifacts at the end of the generated audio sample. This can be done by supplying `pad_end = True` to [`UnivNetFeatureExtractor.__call__`]. See [this issue](https://github.com/seungwonpark/melgan/issues/8) for more details.\n\nUsage Example:\n\n```python\nimport torch\nfrom scipy.io.wavfile import write\nfrom datasets import Audio, load_dataset\n\nfrom transformers import UnivNetFeatureExtractor, UnivNetModel\n\nmodel_id_or_path = \"dg845/univnet-dev\"\nmodel = UnivNetModel.from_pretrained(model_id_or_path)\nfeature_extractor = UnivNetFeatureExtractor.from_pretrained(model_id_or_path)\n\nds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n# Resample the audio to the model and feature extractor's sampling rate.\nds = ds.cast_column(\"audio\", Audio(sampling_rate=feature_extractor.sampling_rate))\n# Pad the end of the converted waveforms to reduce artifacts at the end of the output audio samples.\ninputs = feature_extractor(\n    ds[0][\"audio\"][\"array\"], sampling_rate=ds[0][\"audio\"][\"sampling_rate\"], pad_end=True, return_tensors=\"pt\"\n)\n\nwith torch.no_grad():\n    audio = model(**inputs)\n\n# Remove the extra padding at the end of the output.\naudio = feature_extractor.batch_decode(**audio)[0]\n# Convert to wav file\nwrite(\"sample_audio.wav\", feature_extractor.sampling_rate, audio)\n```\n\nThis model was contributed by [dg845](https://huggingface.co/dg845).\nTo the best of my knowledge, there is no official code release, but an unofficial implementation can be found at [maum-ai/univnet](https://github.com/maum-ai/univnet) with pretrained checkpoints [here](https://github.com/maum-ai/univnet#pre-trained-model).\n\n\n## UnivNetConfig\n\n[[autodoc]] UnivNetConfig\n\n## UnivNetFeatureExtractor\n\n[[autodoc]] UnivNetFeatureExtractor\n    - __call__\n\n## UnivNetModel\n\n[[autodoc]] UnivNetModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 92923bec-8ccb-4e24-9d07-cb1da402e4d9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jukebox", "title": "Jukebox", "tokens": 1168, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"92923bec-8ccb-4e24-9d07-cb1da402e4d9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82432326-652b-42e4-be86-f2571574cb06\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6dd2cc49e774f8ec00d5f65340c3daed7e99db4fcf7440337a0c076d5e37060f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"127f83ac-19ad-4b0d-b5d4-57f1ec1219c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"86d88cefc8b377efbe78c222330b95399004ef1cddafd7550c2f8414715ff1f7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3150, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82432326-652b-42e4-be86-f2571574cb06", "doc_id": "82432326-652b-42e4-be86-f2571574cb06", "ref_doc_id": "82432326-652b-42e4-be86-f2571574cb06", "chroma:document": "# Jukebox\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe Jukebox model was proposed in [Jukebox: A generative model for music](https://arxiv.org/pdf/2005.00341.pdf)\nby Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford,\nIlya Sutskever. It introduces a generative music model which can produce minute long samples that can be conditioned on\nan artist, genres and lyrics.\n\nThe abstract from the paper is the following:\n\n*We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multiscale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers. We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes. We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable. We are releasing thousands of non cherry-picked samples, along with model weights and code.*\n\nAs shown on the following figure, Jukebox is made of 3 `priors` which are decoder only models. They follow the architecture described in [Generating Long Sequences with Sparse Transformers](https://arxiv.org/abs/1904.10509), modified to support longer context length.\nFirst, a autoencoder is used to encode the text lyrics. Next, the first (also called `top_prior`) prior attends to the last hidden states extracted from the lyrics encoder. The priors are linked to the previous priors respectively via an `AudioConditioner` module. The`AudioConditioner` upsamples the outputs of the previous prior to raw tokens at a certain audio frame per second resolution.\nThe metadata such as *artist, genre and timing* are passed to each prior, in the form of a start token and positional embedding for the timing data.  The hidden states are mapped to the closest codebook vector from the VQVAE in order to convert them to raw audio.\n\n![JukeboxModel](https://gist.githubusercontent.com/ArthurZucker/92c1acaae62ebf1b6a951710bdd8b6af/raw/c9c517bf4eff61393f6c7dec9366ef02bdd059a3/jukebox.svg)\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/openai/jukebox).\n\n## Usage tips\n\n- This model only supports inference. This is for a few reasons, mostly because it requires a crazy amount of memory to train. Feel free to open a PR and add what's missing to have a full integration with the hugging face trainer!\n- This model is very slow, and takes 8h to generate a minute long audio using the 5b top prior on a V100 GPU. In order automaticallay handle the device on which the model should execute, use `accelerate`.\n- Contrary to the paper, the order of the priors goes from `0` to `1` as it felt more intuitive : we sample starting from `0`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 127f83ac-19ad-4b0d-b5d4-57f1ec1219c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jukebox", "title": "Jukebox", "tokens": 1168, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"127f83ac-19ad-4b0d-b5d4-57f1ec1219c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82432326-652b-42e4-be86-f2571574cb06\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6dd2cc49e774f8ec00d5f65340c3daed7e99db4fcf7440337a0c076d5e37060f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"92923bec-8ccb-4e24-9d07-cb1da402e4d9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jukebox\", \"title\": \"Jukebox\", \"tokens\": 1168, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"05761f6ff68c6ed193b7b23360a5a97b233c4009b9ccd44d884bc520300d2c96\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1605, \"end_char_idx\": 3972, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82432326-652b-42e4-be86-f2571574cb06", "doc_id": "82432326-652b-42e4-be86-f2571574cb06", "ref_doc_id": "82432326-652b-42e4-be86-f2571574cb06", "chroma:document": "First, a autoencoder is used to encode the text lyrics. Next, the first (also called `top_prior`) prior attends to the last hidden states extracted from the lyrics encoder. The priors are linked to the previous priors respectively via an `AudioConditioner` module. The`AudioConditioner` upsamples the outputs of the previous prior to raw tokens at a certain audio frame per second resolution.\nThe metadata such as *artist, genre and timing* are passed to each prior, in the form of a start token and positional embedding for the timing data.  The hidden states are mapped to the closest codebook vector from the VQVAE in order to convert them to raw audio.\n\n![JukeboxModel](https://gist.githubusercontent.com/ArthurZucker/92c1acaae62ebf1b6a951710bdd8b6af/raw/c9c517bf4eff61393f6c7dec9366ef02bdd059a3/jukebox.svg)\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/openai/jukebox).\n\n## Usage tips\n\n- This model only supports inference. This is for a few reasons, mostly because it requires a crazy amount of memory to train. Feel free to open a PR and add what's missing to have a full integration with the hugging face trainer!\n- This model is very slow, and takes 8h to generate a minute long audio using the 5b top prior on a V100 GPU. In order automaticallay handle the device on which the model should execute, use `accelerate`.\n- Contrary to the paper, the order of the priors goes from `0` to `1` as it felt more intuitive : we sample starting from `0`.\n- Primed sampling (conditioning the sampling on raw audio) requires more memory than ancestral sampling and should be used with `fp16` set to `True`.\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/openai/jukebox).\n\n## JukeboxConfig\n\n[[autodoc]] JukeboxConfig\n\n## JukeboxPriorConfig\n\n[[autodoc]] JukeboxPriorConfig\n\n## JukeboxVQVAEConfig\n\n[[autodoc]] JukeboxVQVAEConfig\n\n## JukeboxTokenizer\n\n[[autodoc]] JukeboxTokenizer\n    - save_vocabulary\n\n## JukeboxModel\n\n[[autodoc]] JukeboxModel\n    - ancestral_sample\n    - primed_sample\n    - continue_sample\n    - upsample\n    - _sample\n\n## JukeboxPrior\n\n[[autodoc]] JukeboxPrior\n    - sample\n    - forward\n\n## JukeboxVQVAE\n\n[[autodoc]] JukeboxVQVAE\n    - forward\n    - encode\n    - decode"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: aa134170-d883-4128-a8a0-fc3d116bed58
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"aa134170-d883-4128-a8a0-fc3d116bed58\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b2ae68c-6161-4ae7-9468-646bbb58fa4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ad26dafdd36c3e046d94623654ea9052ee9e2d778205bcb117b2291ff472048f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0eac1d74-bf4c-42bd-be5c-0b257d66c484\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f03d8a21442981fccacce640760f6c4d570a984c750f06c5c3d1c3bf89c1e9b8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3437, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "ref_doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "chroma:document": "# MusicGen\n\n## Overview\n\nThe MusicGen model was proposed in the paper [Simple and Controllable Music Generation](https://arxiv.org/abs/2306.05284)\nby Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi and Alexandre D\u00e9fossez.\n\nMusicGen is a single stage auto-regressive Transformer model capable of generating high-quality music samples conditioned\non text descriptions or audio prompts. The text descriptions are passed through a frozen text encoder model to obtain a\nsequence of hidden-state representations. MusicGen is then trained to predict discrete audio tokens, or *audio codes*,\nconditioned on these hidden-states. These audio tokens are then decoded using an audio compression model, such as EnCodec,\nto recover the audio waveform.\n\nThrough an efficient token interleaving pattern, MusicGen does not require a self-supervised semantic representation of\nthe text/audio prompts, thus eliminating the need to cascade multiple models to predict a set of codebooks (e.g.\nhierarchically or upsampling). Instead, it is able to generate all the codebooks in a single forward pass.\n\nThe abstract from the paper is the following:\n\n*We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates\nover several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised\nof a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for\ncascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen\ncan generate high-quality samples, while being conditioned on textual description or melodic features, allowing better\ncontrols over the generated output. We conduct extensive empirical evaluation, considering both automatic and human\nstudies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark.\nThrough ablation studies, we shed light over the importance of each of the components comprising MusicGen.*\n\nThis model was contributed by [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found\n[here](https://github.com/facebookresearch/audiocraft). The pre-trained checkpoints can be found on the\n[Hugging Face Hub](https://huggingface.co/models?sort=downloads&search=facebook%2Fmusicgen-).\n\n## Usage tips\n\n- After downloading the original checkpoints from [here](https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md#importing--exporting-models) , you can convert them using the **conversion script** available at\n`src/transformers/models/musicgen/convert_musicgen_transformers.py` with the following command:\n\n```bash\npython src/transformers/models/musicgen/convert_musicgen_transformers.py \\\n    --checkpoint small --pytorch_dump_folder /output/path --safe_serialization \n```\n\n## Generation\n\nMusicGen is compatible with two generation modes: greedy and sampling. In practice, sampling leads to significantly\nbetter results than greedy, thus we encourage sampling mode to be used where possible. Sampling is enabled by default,\nand can be explicitly specified by setting `do_sample=True` in the call to [`MusicgenForConditionalGeneration.generate`],\nor by overriding the model's generation config (see below).\n\nGeneration is limited by the sinusoidal positional embeddings to 30 second inputs."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0eac1d74-bf4c-42bd-be5c-0b257d66c484
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0eac1d74-bf4c-42bd-be5c-0b257d66c484\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b2ae68c-6161-4ae7-9468-646bbb58fa4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ad26dafdd36c3e046d94623654ea9052ee9e2d778205bcb117b2291ff472048f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aa134170-d883-4128-a8a0-fc3d116bed58\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2dee9b830f4101288e91a134ea79d88135ee3ccbe6dfe0760443312ce26e6cdc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6989aa0d-4b75-46db-8217-2c64df70d0da\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b194ce0d6f527ef14b48684edac389226c4638735fbdcc4b72c949f5a0be0b05\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1583, \"end_char_idx\": 5050, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "ref_doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "chroma:document": "Following this approach, we demonstrate how MusicGen\ncan generate high-quality samples, while being conditioned on textual description or melodic features, allowing better\ncontrols over the generated output. We conduct extensive empirical evaluation, considering both automatic and human\nstudies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark.\nThrough ablation studies, we shed light over the importance of each of the components comprising MusicGen.*\n\nThis model was contributed by [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found\n[here](https://github.com/facebookresearch/audiocraft). The pre-trained checkpoints can be found on the\n[Hugging Face Hub](https://huggingface.co/models?sort=downloads&search=facebook%2Fmusicgen-).\n\n## Usage tips\n\n- After downloading the original checkpoints from [here](https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md#importing--exporting-models) , you can convert them using the **conversion script** available at\n`src/transformers/models/musicgen/convert_musicgen_transformers.py` with the following command:\n\n```bash\npython src/transformers/models/musicgen/convert_musicgen_transformers.py \\\n    --checkpoint small --pytorch_dump_folder /output/path --safe_serialization \n```\n\n## Generation\n\nMusicGen is compatible with two generation modes: greedy and sampling. In practice, sampling leads to significantly\nbetter results than greedy, thus we encourage sampling mode to be used where possible. Sampling is enabled by default,\nand can be explicitly specified by setting `do_sample=True` in the call to [`MusicgenForConditionalGeneration.generate`],\nor by overriding the model's generation config (see below).\n\nGeneration is limited by the sinusoidal positional embeddings to 30 second inputs. Meaning, MusicGen cannot generate more\nthan 30 seconds of audio (1503 tokens), and input audio passed by Audio-Prompted Generation contributes to this limit so,\ngiven an input of 20 seconds of audio, MusicGen cannot generate more than 10 seconds of additional audio.\n\nTransformers supports both mono (1-channel) and stereo (2-channel) variants of MusicGen. The mono channel versions \ngenerate a single set of codebooks. The stereo versions generate 2 sets of codebooks, 1 for each channel (left/right), \nand each set of codebooks is decoded independently through the audio compression model. The audio streams for each \nchannel are combined to give the final stereo output.\n\n### Unconditional Generation\n\nThe inputs for unconditional (or 'null') generation can be obtained through the method\n[`MusicgenForConditionalGeneration.get_unconditional_inputs`]:\n\n```python\n>>> from transformers import MusicgenForConditionalGeneration\n\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n>>> unconditional_inputs = model.get_unconditional_inputs(num_samples=1)\n\n>>> audio_values = model.generate(**unconditional_inputs, do_sample=True, max_new_tokens=256)\n```\n\nThe audio outputs are a three-dimensional Torch tensor of shape `(batch_size, num_channels, sequence_length)`. To listen\nto the generated audio samples, you can either play them in an ipynb notebook:\n\n```python\nfrom IPython.display import Audio\n\nsampling_rate = model.config.audio_encoder.sampling_rate\nAudio(audio_values[0].numpy(), rate=sampling_rate)\n```\n\nOr save them as a `.wav` file using a third-party library, e.g."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6989aa0d-4b75-46db-8217-2c64df70d0da
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6989aa0d-4b75-46db-8217-2c64df70d0da\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b2ae68c-6161-4ae7-9468-646bbb58fa4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ad26dafdd36c3e046d94623654ea9052ee9e2d778205bcb117b2291ff472048f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0eac1d74-bf4c-42bd-be5c-0b257d66c484\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"29ea1049f955c271a83ac40883063b57937248a493758e7a451b3a0be38a1b9c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1f8a0adb-f2b3-4800-b544-f0219c774447\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0304c92bc724c8d89aa7d5f8a9491ecd66c6ad340cd4a22e8571d3a60122a890\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3355, \"end_char_idx\": 6624, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "ref_doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "chroma:document": "Generation is limited by the sinusoidal positional embeddings to 30 second inputs. Meaning, MusicGen cannot generate more\nthan 30 seconds of audio (1503 tokens), and input audio passed by Audio-Prompted Generation contributes to this limit so,\ngiven an input of 20 seconds of audio, MusicGen cannot generate more than 10 seconds of additional audio.\n\nTransformers supports both mono (1-channel) and stereo (2-channel) variants of MusicGen. The mono channel versions \ngenerate a single set of codebooks. The stereo versions generate 2 sets of codebooks, 1 for each channel (left/right), \nand each set of codebooks is decoded independently through the audio compression model. The audio streams for each \nchannel are combined to give the final stereo output.\n\n### Unconditional Generation\n\nThe inputs for unconditional (or 'null') generation can be obtained through the method\n[`MusicgenForConditionalGeneration.get_unconditional_inputs`]:\n\n```python\n>>> from transformers import MusicgenForConditionalGeneration\n\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n>>> unconditional_inputs = model.get_unconditional_inputs(num_samples=1)\n\n>>> audio_values = model.generate(**unconditional_inputs, do_sample=True, max_new_tokens=256)\n```\n\nThe audio outputs are a three-dimensional Torch tensor of shape `(batch_size, num_channels, sequence_length)`. To listen\nto the generated audio samples, you can either play them in an ipynb notebook:\n\n```python\nfrom IPython.display import Audio\n\nsampling_rate = model.config.audio_encoder.sampling_rate\nAudio(audio_values[0].numpy(), rate=sampling_rate)\n```\n\nOr save them as a `.wav` file using a third-party library, e.g. `scipy`:\n\n```python\n>>> import scipy\n\n>>> sampling_rate = model.config.audio_encoder.sampling_rate\n>>> scipy.io.wavfile.write(\"musicgen_out.wav\", rate=sampling_rate, data=audio_values[0, 0].numpy())\n```\n\n### Text-Conditional Generation\n\nThe model can generate an audio sample conditioned on a text prompt through use of the [`MusicgenProcessor`] to pre-process\nthe inputs:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-small\")\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> inputs = processor(\n...     text=[\"80s pop track with bassy drums and synth\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe `guidance_scale` is used in classifier free guidance (CFG), setting the weighting between the conditional logits\n(which are predicted from the text prompts) and the unconditional logits (which are predicted from an unconditional or\n'null' prompt). Higher guidance scale encourages the model to generate samples that are more closely linked to the input\nprompt, usually at the expense of poorer audio quality. CFG is enabled by setting `guidance_scale > 1`. For best results,\nuse `guidance_scale=3` (default).\n\n### Audio-Prompted Generation\n\nThe same [`MusicgenProcessor`] can be used to pre-process an audio prompt that is used for audio continuation."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1f8a0adb-f2b3-4800-b544-f0219c774447
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1f8a0adb-f2b3-4800-b544-f0219c774447\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b2ae68c-6161-4ae7-9468-646bbb58fa4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ad26dafdd36c3e046d94623654ea9052ee9e2d778205bcb117b2291ff472048f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6989aa0d-4b75-46db-8217-2c64df70d0da\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6b37e81041c87c5b8a2b5b5d1a879057f3036155bb5af6a74e329ba79ec051cc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"08fecd56-996c-4857-aec0-6d89192faaee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e21ef78a833b3c0679c884f40d690f9f6c1681ba2a6855a70b21a569dbef81e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5051, \"end_char_idx\": 9875, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "ref_doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "chroma:document": "`scipy`:\n\n```python\n>>> import scipy\n\n>>> sampling_rate = model.config.audio_encoder.sampling_rate\n>>> scipy.io.wavfile.write(\"musicgen_out.wav\", rate=sampling_rate, data=audio_values[0, 0].numpy())\n```\n\n### Text-Conditional Generation\n\nThe model can generate an audio sample conditioned on a text prompt through use of the [`MusicgenProcessor`] to pre-process\nthe inputs:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-small\")\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> inputs = processor(\n...     text=[\"80s pop track with bassy drums and synth\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe `guidance_scale` is used in classifier free guidance (CFG), setting the weighting between the conditional logits\n(which are predicted from the text prompts) and the unconditional logits (which are predicted from an unconditional or\n'null' prompt). Higher guidance scale encourages the model to generate samples that are more closely linked to the input\nprompt, usually at the expense of poorer audio quality. CFG is enabled by setting `guidance_scale > 1`. For best results,\nuse `guidance_scale=3` (default).\n\n### Audio-Prompted Generation\n\nThe same [`MusicgenProcessor`] can be used to pre-process an audio prompt that is used for audio continuation. In the\nfollowing example, we load an audio file using the \ud83e\udd17 Datasets library, which can be pip installed through the command\nbelow:\n\n```bash\npip install --upgrade pip\npip install datasets[audio]\n```\n\n```python\n>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-small\")\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> dataset = load_dataset(\"sanchit-gandhi/gtzan\", split=\"train\", streaming=True)\n>>> sample = next(iter(dataset))[\"audio\"]\n\n>>> # take the first half of the audio sample\n>>> sample[\"array\"] = sample[\"array\"][: len(sample[\"array\"]) // 2]\n\n>>> inputs = processor(\n...     audio=sample[\"array\"],\n...     sampling_rate=sample[\"sampling_rate\"],\n...     text=[\"80s blues track with groovy saxophone\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nFor batched audio-prompted generation, the generated `audio_values` can be post-processed to remove padding by using the\n[`MusicgenProcessor`] class:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-small\")\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> dataset = load_dataset(\"sanchit-gandhi/gtzan\", split=\"train\", streaming=True)\n>>> sample = next(iter(dataset))[\"audio\"]\n\n>>> # take the first quarter of the audio sample\n>>> sample_1 = sample[\"array\"][: len(sample[\"array\"]) // 4]\n\n>>> # take the first half of the audio sample\n>>> sample_2 = sample[\"array\"][: len(sample[\"array\"]) // 2]\n\n>>> inputs = processor(\n...     audio=[sample_1, sample_2],\n...     sampling_rate=sample[\"sampling_rate\"],\n...     text=[\"80s blues track with groovy saxophone\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n\n>>> # post-process to remove padding from the batched audio\n>>> audio_values = processor.batch_decode(audio_values, padding_mask=inputs.padding_mask)\n```\n\n### Generation Configuration\n\nThe default parameters that control the generation process, such as sampling, guidance scale and number of generated \ntokens, can be found in the model's generation config, and updated as desired:\n\n```python\n>>> from transformers import MusicgenForConditionalGeneration\n\n>>> model = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\")\n\n>>> # inspect the default generation config\n>>> model.generation_config\n\n>>> # increase the guidance scale to 4.0\n>>> model.generation_config.guidance_scale = 4.0\n\n>>> # decrease the max length to 256 tokens\n>>> model.generation_config.max_length = 256\n```\n\nNote that any arguments passed to the generate method will **supersede** those in the generation config, so setting \n`do_sample=False` in the call to generate will supersede the setting of `model.generation_config.do_sample` in the \ngeneration config."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 08fecd56-996c-4857-aec0-6d89192faaee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen", "title": "MusicGen", "tokens": 2891, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"08fecd56-996c-4857-aec0-6d89192faaee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b2ae68c-6161-4ae7-9468-646bbb58fa4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ad26dafdd36c3e046d94623654ea9052ee9e2d778205bcb117b2291ff472048f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1f8a0adb-f2b3-4800-b544-f0219c774447\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen\", \"title\": \"MusicGen\", \"tokens\": 2891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"44b2f8a3f1b9758d6bc85761b34b1fd0e840ec279604946f6d59ccd0e27e3d4c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9877, \"end_char_idx\": 12541, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "ref_doc_id": "2b2ae68c-6161-4ae7-9468-646bbb58fa4d", "chroma:document": "## Model Structure\n\nThe MusicGen model can be de-composed into three distinct stages:\n1. Text encoder: maps the text inputs to a sequence of hidden-state representations. The pre-trained MusicGen models use a frozen text encoder from either T5 or Flan-T5\n2. MusicGen decoder: a language model (LM) that auto-regressively generates audio tokens (or codes) conditional on the encoder hidden-state representations\n3. Audio encoder/decoder: used to encode an audio prompt to use as prompt tokens, and recover the audio waveform from the audio tokens predicted by the decoder\n\nThus, the MusicGen model can either be used as a standalone decoder model, corresponding to the class [`MusicgenForCausalLM`],\nor as a composite model that includes the text encoder and audio encoder/decoder, corresponding to the class\n[`MusicgenForConditionalGeneration`]. If only the decoder needs to be loaded from the pre-trained checkpoint, it can be loaded by first \nspecifying the correct config, or be accessed through the `.decoder` attribute of the composite model:\n\n```python\n>>> from transformers import AutoConfig, MusicgenForCausalLM, MusicgenForConditionalGeneration\n\n>>> # Option 1: get decoder config and pass to `.from_pretrained`\n>>> decoder_config = AutoConfig.from_pretrained(\"facebook/musicgen-small\").decoder\n>>> decoder = MusicgenForCausalLM.from_pretrained(\"facebook/musicgen-small\", **decoder_config)\n\n>>> # Option 2: load the entire composite model, but only return the decoder\n>>> decoder = MusicgenForConditionalGeneration.from_pretrained(\"facebook/musicgen-small\").decoder\n```\n\nSince the text encoder and audio encoder/decoder models are frozen during training, the MusicGen decoder [`MusicgenForCausalLM`]\ncan be trained standalone on a dataset of encoder hidden-states and audio codes. For inference, the trained decoder can\nbe combined with the frozen text encoder and audio encoder/decoders to recover the composite [`MusicgenForConditionalGeneration`]\nmodel.\n\nTips:\n* MusicGen is trained on the 32kHz checkpoint of Encodec. You should ensure you use a compatible version of the Encodec model.\n* Sampling mode tends to deliver better results than greedy - you can toggle sampling with the variable `do_sample` in the call to [`MusicgenForConditionalGeneration.generate`]\n\n## MusicgenDecoderConfig\n\n[[autodoc]] MusicgenDecoderConfig\n\n## MusicgenConfig\n\n[[autodoc]] MusicgenConfig\n\n## MusicgenProcessor\n\n[[autodoc]] MusicgenProcessor\n\n## MusicgenModel\n\n[[autodoc]] MusicgenModel\n    - forward\n\n## MusicgenForCausalLM\n\n[[autodoc]] MusicgenForCausalLM\n    - forward\n\n## MusicgenForConditionalGeneration\n\n[[autodoc]] MusicgenForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5bd70dbe-7ca5-4ed9-916f-f0635eebf21d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swin", "title": "Swin Transformer", "tokens": 1163, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5bd70dbe-7ca5-4ed9-916f-f0635eebf21d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c624cf7e-d39d-4ae4-b722-57269428ff71\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6fbc14a33f7ba96d2d6accd5077b047e156bba49e35ae2cef2f811d8e53bce0c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8705a2c2-04cd-469d-a087-7c298c866288\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6cc4e1c3227f42ca5052ed670afcacce19990da0626534ab887f1f49701ebf5e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3191, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c624cf7e-d39d-4ae4-b722-57269428ff71", "doc_id": "c624cf7e-d39d-4ae4-b722-57269428ff71", "ref_doc_id": "c624cf7e-d39d-4ae4-b722-57269428ff71", "chroma:document": "# Swin Transformer\n\n## Overview\n\nThe Swin Transformer was proposed in [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://arxiv.org/abs/2103.14030)\nby Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo.\n\nThe abstract from the paper is the following:\n\n*This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone\nfor computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains,\nsuch as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text.\nTo address these differences, we propose a hierarchical Transformer whose representation is computed with \\bold{S}hifted\n\\bold{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping\nlocal windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at\nvarious scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it\ncompatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense\nprediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation\n(53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and\n+2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones.\nThe hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/swin_transformer_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Swin Transformer architecture. Taken from the <a href=\"https://arxiv.org/abs/2102.03334\">original paper</a>.</small>\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The Tensorflow version of this model was contributed by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/microsoft/Swin-Transformer).\n\n## Usage tips\n\n- Swin pads the inputs supporting any input height and width (if divisible by `32`).\n- Swin can be used as a *backbone*. When `output_hidden_states = True`, it will output both `hidden_states` and `reshaped_hidden_states`. The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than `(batch_size, sequence_length, num_channels)`.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Swin Transformer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`SwinForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8705a2c2-04cd-469d-a087-7c298c866288
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swin", "title": "Swin Transformer", "tokens": 1163, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8705a2c2-04cd-469d-a087-7c298c866288\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c624cf7e-d39d-4ae4-b722-57269428ff71\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6fbc14a33f7ba96d2d6accd5077b047e156bba49e35ae2cef2f811d8e53bce0c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5bd70dbe-7ca5-4ed9-916f-f0635eebf21d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swin\", \"title\": \"Swin Transformer\", \"tokens\": 1163, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37a45ec63b9e88cebc4a89fe20e1bb1733e0faa14e6d0fc0cff23561855350ec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1683, \"end_char_idx\": 4248, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c624cf7e-d39d-4ae4-b722-57269428ff71", "doc_id": "c624cf7e-d39d-4ae4-b722-57269428ff71", "ref_doc_id": "c624cf7e-d39d-4ae4-b722-57269428ff71", "chroma:document": "The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/swin_transformer_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Swin Transformer architecture. Taken from the <a href=\"https://arxiv.org/abs/2102.03334\">original paper</a>.</small>\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The Tensorflow version of this model was contributed by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/microsoft/Swin-Transformer).\n\n## Usage tips\n\n- Swin pads the inputs supporting any input height and width (if divisible by `32`).\n- Swin can be used as a *backbone*. When `output_hidden_states = True`, it will output both `hidden_states` and `reshaped_hidden_states`. The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than `(batch_size, sequence_length, num_channels)`.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Swin Transformer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`SwinForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nBesides that:\n\n- [`SwinForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## SwinConfig\n\n[[autodoc]] SwinConfig\n\n<frameworkcontent>\n<pt>\n\n## SwinModel\n\n[[autodoc]] SwinModel\n    - forward\n\n## SwinForMaskedImageModeling\n\n[[autodoc]] SwinForMaskedImageModeling\n    - forward\n\n## SwinForImageClassification\n\n[[autodoc]] transformers.SwinForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFSwinModel\n\n[[autodoc]] TFSwinModel\n    - call\n\n## TFSwinForMaskedImageModeling\n\n[[autodoc]] TFSwinForMaskedImageModeling\n    - call\n\n## TFSwinForImageClassification\n\n[[autodoc]] transformers.TFSwinForImageClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 127c8902-1b1f-4840-9369-b2aac1f93d55
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/perceiver", "title": "Perceiver", "tokens": 2357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"127c8902-1b1f-4840-9369-b2aac1f93d55\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eebd4b6b-af7f-4f9d-91cc-47497bab4b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fdc324970717e9e698d74b0199b29841740e5550b79cddc4f6f1c26466df61e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b9dd2a36-037f-4b72-8bb1-37e2c4f36639\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a9da266049e34d89d7efd20be55aca38b1bd8bde87ef72eda0f796bc53d20c68\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3415, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "doc_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "ref_doc_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "chroma:document": "# Perceiver\n\n## Overview\n\nThe Perceiver IO model was proposed in [Perceiver IO: A General Architecture for Structured Inputs &\nOutputs](https://arxiv.org/abs/2107.14795) by Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch,\nCatalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier H\u00e9naff, Matthew M.\nBotvinick, Andrew Zisserman, Oriol Vinyals, Jo\u00e3o Carreira.\n\nPerceiver IO is a generalization of [Perceiver](https://arxiv.org/abs/2103.03206) to handle arbitrary outputs in\naddition to arbitrary inputs. The original Perceiver only produced a single classification label. In addition to\nclassification labels, Perceiver IO can produce (for example) language, optical flow, and multimodal videos with audio.\nThis is done using the same building blocks as the original Perceiver. The computational complexity of Perceiver IO is\nlinear in the input and output size and the bulk of the processing occurs in the latent space, allowing us to process\ninputs and outputs that are much larger than can be handled by standard Transformers. This means, for example,\nPerceiver IO can do BERT-style masked language modeling directly using bytes instead of tokenized inputs.\n\nThe abstract from the paper is the following:\n\n*The recently-proposed Perceiver model obtains good results on several domains (images, audio, multimodal, point\nclouds) while scaling linearly in compute and memory with the input size. While the Perceiver supports many kinds of\ninputs, it can only produce very simple outputs such as class scores. Perceiver IO overcomes this limitation without\nsacrificing the original's appealing properties by learning to flexibly query the model's latent space to produce\noutputs of arbitrary size and semantics. Perceiver IO still decouples model depth from data size and still scales\nlinearly with data size, but now with respect to both input and output sizes. The full Perceiver IO model achieves\nstrong results on tasks with highly structured output spaces, such as natural language and visual understanding,\nStarCraft II, and multi-task and multi-modal domains. As highlights, Perceiver IO matches a Transformer-based BERT\nbaseline on the GLUE language benchmark without the need for input tokenization and achieves state-of-the-art\nperformance on Sintel optical flow estimation.*\n\nHere's a TLDR explaining how Perceiver works:\n\nThe main problem with the self-attention mechanism of the Transformer is that the time and memory requirements scale\nquadratically with the sequence length. Hence, models like BERT and RoBERTa are limited to a max sequence length of 512\ntokens. Perceiver aims to solve this issue by, instead of performing self-attention on the inputs, perform it on a set\nof latent variables, and only use the inputs for cross-attention. In this way, the time and memory requirements don't\ndepend on the length of the inputs anymore, as one uses a fixed amount of latent variables, like 256 or 512. These are\nrandomly initialized, after which they are trained end-to-end using backpropagation.\n\nInternally, [`PerceiverModel`] will create the latents, which is a tensor of shape `(batch_size, num_latents,\nd_latents)`. One must provide `inputs` (which could be text, images, audio, you name it!) to the model, which it will\nuse to perform cross-attention with the latents. The output of the Perceiver encoder is a tensor of the same shape."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b9dd2a36-037f-4b72-8bb1-37e2c4f36639
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/perceiver", "title": "Perceiver", "tokens": 2357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b9dd2a36-037f-4b72-8bb1-37e2c4f36639\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eebd4b6b-af7f-4f9d-91cc-47497bab4b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fdc324970717e9e698d74b0199b29841740e5550b79cddc4f6f1c26466df61e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"127c8902-1b1f-4840-9369-b2aac1f93d55\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1cb390c4539e0edc4f097bc902870866dfcc2a99cb8e5ae6efc06585a41f6c20\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5f25d433-2406-46fb-8235-bacfb2ac90a9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9c6db76afe5f6d0459b4ce4d5254b36624ff5cc437724e9241318bb6316e3196\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1771, \"end_char_idx\": 5123, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "doc_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "ref_doc_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "chroma:document": "Perceiver IO still decouples model depth from data size and still scales\nlinearly with data size, but now with respect to both input and output sizes. The full Perceiver IO model achieves\nstrong results on tasks with highly structured output spaces, such as natural language and visual understanding,\nStarCraft II, and multi-task and multi-modal domains. As highlights, Perceiver IO matches a Transformer-based BERT\nbaseline on the GLUE language benchmark without the need for input tokenization and achieves state-of-the-art\nperformance on Sintel optical flow estimation.*\n\nHere's a TLDR explaining how Perceiver works:\n\nThe main problem with the self-attention mechanism of the Transformer is that the time and memory requirements scale\nquadratically with the sequence length. Hence, models like BERT and RoBERTa are limited to a max sequence length of 512\ntokens. Perceiver aims to solve this issue by, instead of performing self-attention on the inputs, perform it on a set\nof latent variables, and only use the inputs for cross-attention. In this way, the time and memory requirements don't\ndepend on the length of the inputs anymore, as one uses a fixed amount of latent variables, like 256 or 512. These are\nrandomly initialized, after which they are trained end-to-end using backpropagation.\n\nInternally, [`PerceiverModel`] will create the latents, which is a tensor of shape `(batch_size, num_latents,\nd_latents)`. One must provide `inputs` (which could be text, images, audio, you name it!) to the model, which it will\nuse to perform cross-attention with the latents. The output of the Perceiver encoder is a tensor of the same shape. One\ncan then, similar to BERT, convert the last hidden states of the latents to classification logits by averaging along\nthe sequence dimension, and placing a linear layer on top of that to project the `d_latents` to `num_labels`.\n\nThis was the idea of the original Perceiver paper. However, it could only output classification logits. In a follow-up\nwork, PerceiverIO, they generalized it to let the model also produce outputs of arbitrary size. How, you might ask? The\nidea is actually relatively simple: one defines outputs of an arbitrary size, and then applies cross-attention with the\nlast hidden states of the latents, using the outputs as queries, and the latents as keys and values.\n\nSo let's say one wants to perform masked language modeling (BERT-style) with the Perceiver. As the Perceiver's input\nlength will not have an impact on the computation time of the self-attention layers, one can provide raw bytes,\nproviding `inputs` of length 2048 to the model. If one now masks out certain of these 2048 tokens, one can define the\n`outputs` as being of shape: `(batch_size, 2048, 768)`. Next, one performs cross-attention with the final hidden states\nof the latents to update the `outputs` tensor. After cross-attention, one still has a tensor of shape `(batch_size,\n2048, 768)`. One can then place a regular language modeling head on top, to project the last dimension to the\nvocabulary size of the model, i.e. creating logits of shape `(batch_size, 2048, 262)` (as Perceiver uses a vocabulary\nsize of 262 byte IDs).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perceiver_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Perceiver IO architecture."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5f25d433-2406-46fb-8235-bacfb2ac90a9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/perceiver", "title": "Perceiver", "tokens": 2357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5f25d433-2406-46fb-8235-bacfb2ac90a9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eebd4b6b-af7f-4f9d-91cc-47497bab4b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fdc324970717e9e698d74b0199b29841740e5550b79cddc4f6f1c26466df61e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b9dd2a36-037f-4b72-8bb1-37e2c4f36639\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5be9dbaad345333bf304f8753a1b2bcb3ccbd54d1fd821aabba1727c12998a8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fcb2a903-c47e-463c-aeae-2e400c272720\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7ae2dbbc2c922f8330c89be1b3c6a7da4bcb4699b6b8d01282c1963b79175f6b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3648, \"end_char_idx\": 6177, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "doc_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "ref_doc_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "chroma:document": "This was the idea of the original Perceiver paper. However, it could only output classification logits. In a follow-up\nwork, PerceiverIO, they generalized it to let the model also produce outputs of arbitrary size. How, you might ask? The\nidea is actually relatively simple: one defines outputs of an arbitrary size, and then applies cross-attention with the\nlast hidden states of the latents, using the outputs as queries, and the latents as keys and values.\n\nSo let's say one wants to perform masked language modeling (BERT-style) with the Perceiver. As the Perceiver's input\nlength will not have an impact on the computation time of the self-attention layers, one can provide raw bytes,\nproviding `inputs` of length 2048 to the model. If one now masks out certain of these 2048 tokens, one can define the\n`outputs` as being of shape: `(batch_size, 2048, 768)`. Next, one performs cross-attention with the final hidden states\nof the latents to update the `outputs` tensor. After cross-attention, one still has a tensor of shape `(batch_size,\n2048, 768)`. One can then place a regular language modeling head on top, to project the last dimension to the\nvocabulary size of the model, i.e. creating logits of shape `(batch_size, 2048, 262)` (as Perceiver uses a vocabulary\nsize of 262 byte IDs).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perceiver_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Perceiver IO architecture. Taken from the <a href=\"https://arxiv.org/abs/2105.15203\">original paper</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/deepmind/deepmind-research/tree/master/perceiver).\n\n<Tip warning={true}>\n\nPerceiver does **not** work with `torch.nn.DataParallel` due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035)\n\n</Tip>\n\n## Resources\n\n- The quickest way to get started with the Perceiver is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Perceiver).\n- Refer to the [blog post](https://huggingface.co/blog/perceiver) if you want to fully understand how the model works and\nis implemented in the library. Note that the models available in the library only showcase some examples of what you can do\nwith the Perceiver. There are many more use cases, including question answering, named-entity recognition, object detection,\naudio classification, video classification, etc."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fcb2a903-c47e-463c-aeae-2e400c272720
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/perceiver", "title": "Perceiver", "tokens": 2357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fcb2a903-c47e-463c-aeae-2e400c272720\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eebd4b6b-af7f-4f9d-91cc-47497bab4b82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fdc324970717e9e698d74b0199b29841740e5550b79cddc4f6f1c26466df61e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5f25d433-2406-46fb-8235-bacfb2ac90a9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/perceiver\", \"title\": \"Perceiver\", \"tokens\": 2357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"508303011e24436298f643ed8f8f74bf4d5271e3d788c44b7d1d5704ef3e9e68\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4623, \"end_char_idx\": 9435, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "doc_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "ref_doc_id": "eebd4b6b-af7f-4f9d-91cc-47497bab4b82", "chroma:document": "After cross-attention, one still has a tensor of shape `(batch_size,\n2048, 768)`. One can then place a regular language modeling head on top, to project the last dimension to the\nvocabulary size of the model, i.e. creating logits of shape `(batch_size, 2048, 262)` (as Perceiver uses a vocabulary\nsize of 262 byte IDs).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perceiver_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Perceiver IO architecture. Taken from the <a href=\"https://arxiv.org/abs/2105.15203\">original paper</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/deepmind/deepmind-research/tree/master/perceiver).\n\n<Tip warning={true}>\n\nPerceiver does **not** work with `torch.nn.DataParallel` due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035)\n\n</Tip>\n\n## Resources\n\n- The quickest way to get started with the Perceiver is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Perceiver).\n- Refer to the [blog post](https://huggingface.co/blog/perceiver) if you want to fully understand how the model works and\nis implemented in the library. Note that the models available in the library only showcase some examples of what you can do\nwith the Perceiver. There are many more use cases, including question answering, named-entity recognition, object detection,\naudio classification, video classification, etc.\n- [Text classification task guide](../tasks/sequence_classification)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Image classification task guide](../tasks/image_classification)\n\n## Perceiver specific outputs\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverModelOutput\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverDecoderOutput\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverMaskedLMOutput\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverClassifierOutput\n\n## PerceiverConfig\n\n[[autodoc]] PerceiverConfig\n\n## PerceiverTokenizer\n\n[[autodoc]] PerceiverTokenizer\n    - __call__\n\n## PerceiverFeatureExtractor\n\n[[autodoc]] PerceiverFeatureExtractor\n    - __call__\n\n## PerceiverImageProcessor\n\n[[autodoc]] PerceiverImageProcessor\n    - preprocess\n\n## PerceiverTextPreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverTextPreprocessor\n\n## PerceiverImagePreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverImagePreprocessor\n\n## PerceiverOneHotPreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverOneHotPreprocessor\n\n## PerceiverAudioPreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverAudioPreprocessor\n\n## PerceiverMultimodalPreprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverMultimodalPreprocessor\n\n## PerceiverProjectionDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverProjectionDecoder\n\n## PerceiverBasicDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverBasicDecoder\n\n## PerceiverClassificationDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverClassificationDecoder\n\n## PerceiverOpticalFlowDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverOpticalFlowDecoder\n\n## PerceiverBasicVideoAutoencodingDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverBasicVideoAutoencodingDecoder\n\n## PerceiverMultimodalDecoder\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverMultimodalDecoder\n\n## PerceiverProjectionPostprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverProjectionPostprocessor\n\n## PerceiverAudioPostprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverAudioPostprocessor\n\n## PerceiverClassificationPostprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverClassificationPostprocessor\n\n## PerceiverMultimodalPostprocessor\n\n[[autodoc]] models.perceiver.modeling_perceiver.PerceiverMultimodalPostprocessor\n\n## PerceiverModel\n\n[[autodoc]] PerceiverModel\n    - forward\n\n## PerceiverForMaskedLM\n\n[[autodoc]] PerceiverForMaskedLM\n    - forward\n\n## PerceiverForSequenceClassification\n\n[[autodoc]] PerceiverForSequenceClassification\n    - forward\n\n## PerceiverForImageClassificationLearned\n\n[[autodoc]] PerceiverForImageClassificationLearned\n    - forward\n\n## PerceiverForImageClassificationFourier\n\n[[autodoc]] PerceiverForImageClassificationFourier\n    - forward\n\n## PerceiverForImageClassificationConvProcessing\n\n[[autodoc]] PerceiverForImageClassificationConvProcessing\n    - forward\n\n## PerceiverForOpticalFlow\n\n[[autodoc]] PerceiverForOpticalFlow\n    - forward\n\n## PerceiverForMultimodalAutoencoding\n\n[[autodoc]] PerceiverForMultimodalAutoencoding\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b112d6db-2487-4911-8a99-c9dfaa62efb7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xmod", "title": "X-MOD", "tokens": 1257, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b112d6db-2487-4911-8a99-c9dfaa62efb7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"da849b83-9cc4-410c-b713-2e18d0299e38\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bc6e9628f5091d0f2f591e5b53c0a8ad9908b26365ca14f0cf453eeb49649e62\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"941daf40-f743-4e29-8c21-9f726e3449fe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bff30ca492a236fc0a5849542af0e64ee2481ff8dfe39d3cdca96f6e6d1f4f65\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3012, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "da849b83-9cc4-410c-b713-2e18d0299e38", "doc_id": "da849b83-9cc4-410c-b713-2e18d0299e38", "ref_doc_id": "da849b83-9cc4-410c-b713-2e18d0299e38", "chroma:document": "# X-MOD\n\n## Overview\n\nThe X-MOD model was proposed in [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](http://dx.doi.org/10.18653/v1/2022.naacl-main.255) by Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe.\nX-MOD extends multilingual masked language models like [XLM-R](xlm-roberta) to include language-specific modular components (_language adapters_) during pre-training. For fine-tuning, the language adapters in each transformer layer are frozen.\n\nThe abstract from the paper is the following:\n\n*Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-MOD) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.*\n\nThis model was contributed by [jvamvas](https://huggingface.co/jvamvas).\nThe original code can be found [here](https://github.com/facebookresearch/fairseq/tree/58cc6cca18f15e6d56e3f60c959fe4f878960a60/fairseq/models/xmod) and the original documentation is found [here](https://github.com/facebookresearch/fairseq/tree/58cc6cca18f15e6d56e3f60c959fe4f878960a60/examples/xmod).\n\n## Usage tips\n\nTips:\n- X-MOD is similar to [XLM-R](xlm-roberta), but a difference is that the input language needs to be specified so that the correct language adapter can be activated.\n- The main models \u2013 base and large \u2013 have adapters for 81 languages.\n\n## Adapter Usage\n\n### Input language\n\nThere are two ways to specify the input language:\n1. By setting a default language before using the model:\n\n```python\nfrom transformers import XmodModel\n\nmodel = XmodModel.from_pretrained(\"facebook/xmod-base\")\nmodel.set_default_language(\"en_XX\")\n```\n\n2. By explicitly passing the index of the language adapter for each sample:\n\n```python\nimport torch\n\ninput_ids = torch.tensor(\n    [\n        [0, 581, 10269, 83, 99942, 136, 60742, 23, 70, 80583, 18276, 2],\n        [0, 1310, 49083, 443, 269, 71, 5486, 165, 60429, 660, 23, 2],\n    ]\n)\nlang_ids = torch.LongTensor(\n    [\n        0,  # en_XX\n        8,  # de_DE\n    ]\n)\noutput = model(input_ids, lang_ids=lang_ids)\n```\n\n### Fine-tuning\nThe paper recommends that the embedding layer and the language adapters are frozen during fine-tuning."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 941daf40-f743-4e29-8c21-9f726e3449fe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xmod", "title": "X-MOD", "tokens": 1257, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"941daf40-f743-4e29-8c21-9f726e3449fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"da849b83-9cc4-410c-b713-2e18d0299e38\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bc6e9628f5091d0f2f591e5b53c0a8ad9908b26365ca14f0cf453eeb49649e62\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b112d6db-2487-4911-8a99-c9dfaa62efb7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xmod\", \"title\": \"X-MOD\", \"tokens\": 1257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b17bcf11818db0b024a2d9efcbcf6419119b56f7fe6ab5644ffda760ccdbf597\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1932, \"end_char_idx\": 4348, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "da849b83-9cc4-410c-b713-2e18d0299e38", "doc_id": "da849b83-9cc4-410c-b713-2e18d0299e38", "ref_doc_id": "da849b83-9cc4-410c-b713-2e18d0299e38", "chroma:document": "## Usage tips\n\nTips:\n- X-MOD is similar to [XLM-R](xlm-roberta), but a difference is that the input language needs to be specified so that the correct language adapter can be activated.\n- The main models \u2013 base and large \u2013 have adapters for 81 languages.\n\n## Adapter Usage\n\n### Input language\n\nThere are two ways to specify the input language:\n1. By setting a default language before using the model:\n\n```python\nfrom transformers import XmodModel\n\nmodel = XmodModel.from_pretrained(\"facebook/xmod-base\")\nmodel.set_default_language(\"en_XX\")\n```\n\n2. By explicitly passing the index of the language adapter for each sample:\n\n```python\nimport torch\n\ninput_ids = torch.tensor(\n    [\n        [0, 581, 10269, 83, 99942, 136, 60742, 23, 70, 80583, 18276, 2],\n        [0, 1310, 49083, 443, 269, 71, 5486, 165, 60429, 660, 23, 2],\n    ]\n)\nlang_ids = torch.LongTensor(\n    [\n        0,  # en_XX\n        8,  # de_DE\n    ]\n)\noutput = model(input_ids, lang_ids=lang_ids)\n```\n\n### Fine-tuning\nThe paper recommends that the embedding layer and the language adapters are frozen during fine-tuning. A method for doing this is provided:\n\n```python\nmodel.freeze_embeddings_and_language_adapters()\n# Fine-tune the model ...\n```\n\n### Cross-lingual transfer\nAfter fine-tuning, zero-shot cross-lingual transfer can be tested by activating the language adapter of the target language:\n\n```python\nmodel.set_default_language(\"de_DE\")\n# Evaluate the model on German examples ...\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## XmodConfig\n\n[[autodoc]] XmodConfig\n\n## XmodModel\n\n[[autodoc]] XmodModel\n    - forward\n\n## XmodForCausalLM\n\n[[autodoc]] XmodForCausalLM\n    - forward\n\n## XmodForMaskedLM\n\n[[autodoc]] XmodForMaskedLM\n    - forward\n\n## XmodForSequenceClassification\n\n[[autodoc]] XmodForSequenceClassification\n    - forward\n\n## XmodForMultipleChoice\n\n[[autodoc]] XmodForMultipleChoice\n    - forward\n\n## XmodForTokenClassification\n\n[[autodoc]] XmodForTokenClassification\n    - forward\n\n## XmodForQuestionAnswering\n\n[[autodoc]] XmodForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f04363a2-e157-4c89-af18-1bc632088493
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/distilbert", "title": "DistilBERT", "tokens": 3717, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f04363a2-e157-4c89-af18-1bc632088493\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef76daf8-0690-4ee7-b552-3927bdd35fb9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3871d1613d76040ce641a5ddd524847b969eb92bcc98223c068ab18273b4dc80\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0881e485-f2d3-41c1-983e-6941957353aa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"04dcd922dca7a9bee23ceed40094064c386afeb924801d6aa7f5d63299c50d9a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3097, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "ref_doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "chroma:document": "# DistilBERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=distilbert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-distilbert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/distilbert-base-uncased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n<a href=\"https://huggingface.co/papers/1910.01108\">\n<img alt=\"Paper page\" src=\"https://img.shields.io/badge/Paper%20page-1910.01108-green\">\n</a>\n</div>\n\n## Overview\n\nThe DistilBERT model was proposed in the blog post [Smaller, faster, cheaper, lighter: Introducing DistilBERT, a\ndistilled version of BERT](https://medium.com/huggingface/distilbert-8cf3380435b5), and the paper [DistilBERT, a\ndistilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108). DistilBERT is a\nsmall, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than\n*google-bert/bert-base-uncased*, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language\nunderstanding benchmark.\n\nThe abstract from the paper is the following:\n\n*As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP),\noperating these large models in on-the-edge and/or under constrained computational training or inference budgets\nremains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation\nmodel, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger\ncounterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage\nknowledge distillation during the pretraining phase and show that it is possible to reduce the size of a BERT model by\n40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive\nbiases learned by larger models during pretraining, we introduce a triple loss combining language modeling,\ndistillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we\ndemonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device\nstudy.*\n\nThis model was contributed by [victorsanh](https://huggingface.co/victorsanh). This model jax version was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation).\n\n## Usage tips\n\n- DistilBERT doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just\n  separate your segments with the separation token `tokenizer.sep_token` (or `[SEP]`).\n- DistilBERT doesn't have options to select the input positions (`position_ids` input). This could be added if\n  necessary though, just let us know if you need this option.\n- Same as BERT but smaller."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0881e485-f2d3-41c1-983e-6941957353aa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/distilbert", "title": "DistilBERT", "tokens": 3717, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0881e485-f2d3-41c1-983e-6941957353aa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef76daf8-0690-4ee7-b552-3927bdd35fb9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3871d1613d76040ce641a5ddd524847b969eb92bcc98223c068ab18273b4dc80\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f04363a2-e157-4c89-af18-1bc632088493\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ee363f7c35f04dac0e40157180fc75fa94769f7c43753942ab9626895f55cd47\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"270c7f7f-f959-402b-95e8-3d4c8e3ecddd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bdec7f48732a2add9732aec2d15dee1ec94fbec1a6410808cfafb3e6d761313a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1463, \"end_char_idx\": 3500, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "ref_doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "chroma:document": "In this work, we propose a method to pre-train a smaller general-purpose language representation\nmodel, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger\ncounterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage\nknowledge distillation during the pretraining phase and show that it is possible to reduce the size of a BERT model by\n40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive\nbiases learned by larger models during pretraining, we introduce a triple loss combining language modeling,\ndistillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we\ndemonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device\nstudy.*\n\nThis model was contributed by [victorsanh](https://huggingface.co/victorsanh). This model jax version was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation).\n\n## Usage tips\n\n- DistilBERT doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just\n  separate your segments with the separation token `tokenizer.sep_token` (or `[SEP]`).\n- DistilBERT doesn't have options to select the input positions (`position_ids` input). This could be added if\n  necessary though, just let us know if you need this option.\n- Same as BERT but smaller. Trained by distillation of the pretrained BERT model, meaning it\u2019s been trained to predict the same probabilities as the larger model. The actual objective is a combination of:\n\n    * finding the same probabilities as the teacher model\n    * predicting the masked tokens correctly (but no next-sentence objective)\n    * a cosine similarity between the hidden states of the student and the teacher model"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 270c7f7f-f959-402b-95e8-3d4c8e3ecddd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/distilbert", "title": "DistilBERT", "tokens": 3717, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"270c7f7f-f959-402b-95e8-3d4c8e3ecddd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef76daf8-0690-4ee7-b552-3927bdd35fb9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3871d1613d76040ce641a5ddd524847b969eb92bcc98223c068ab18273b4dc80\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0881e485-f2d3-41c1-983e-6941957353aa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0fc5faf2f2f68da6c40dc832dc0a3763b7633d22a1423c928687cbaae006e601\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e478ee79-8414-438a-a4c2-cf7f70d915d2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f26a3e68e09c839af10cbd22af4ae5ee14d71f670541aaadf072dc37904e19c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1693, \"end_char_idx\": 6039, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "ref_doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "chroma:document": "While most prior work investigated the use of distillation for building task-specific models, we leverage\nknowledge distillation during the pretraining phase and show that it is possible to reduce the size of a BERT model by\n40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive\nbiases learned by larger models during pretraining, we introduce a triple loss combining language modeling,\ndistillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we\ndemonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device\nstudy.*\n\nThis model was contributed by [victorsanh](https://huggingface.co/victorsanh). This model jax version was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation).\n\n## Usage tips\n\n- DistilBERT doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just\n  separate your segments with the separation token `tokenizer.sep_token` (or `[SEP]`).\n- DistilBERT doesn't have options to select the input positions (`position_ids` input). This could be added if\n  necessary though, just let us know if you need this option.\n- Same as BERT but smaller. Trained by distillation of the pretrained BERT model, meaning it\u2019s been trained to predict the same probabilities as the larger model. The actual objective is a combination of:\n\n    * finding the same probabilities as the teacher model\n    * predicting the masked tokens correctly (but no next-sentence objective)\n    * a cosine similarity between the hidden states of the student and the teacher model\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DistilBERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [Getting Started with Sentiment Analysis using Python](https://huggingface.co/blog/sentiment-analysis-python) with DistilBERT.\n- A blog post on how to [train DistilBERT with Blurr for sequence classification](https://huggingface.co/blog/fastai).\n- A blog post on how to use [Ray to tune DistilBERT hyperparameters](https://huggingface.co/blog/ray-tune).\n- A blog post on how to [train DistilBERT with Hugging Face and Amazon SageMaker](https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face).\n- A notebook on how to [finetune DistilBERT for multi-label classification](https://colab.research.google.com/github/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune DistilBERT for multiclass classification with PyTorch](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune DistilBERT for text classification in TensorFlow](https://colab.research.google.com/github/peterbayerle/huggingface_notebook/blob/main/distilbert_tf.ipynb). \ud83c\udf0e\n- [`DistilBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFDistilBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).\n- [`FlaxDistilBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e478ee79-8414-438a-a4c2-cf7f70d915d2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/distilbert", "title": "DistilBERT", "tokens": 3717, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e478ee79-8414-438a-a4c2-cf7f70d915d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef76daf8-0690-4ee7-b552-3927bdd35fb9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3871d1613d76040ce641a5ddd524847b969eb92bcc98223c068ab18273b4dc80\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"270c7f7f-f959-402b-95e8-3d4c8e3ecddd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7c6c73398b095198fff4ac6c02aef4a7e54c7837fd0cb9aadfa419cef718ef90\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aae41262-a6db-4cd4-acb1-47fee0fa5026\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b98223965a68643584cc0519f9058d552be133147aa0c344798449faa88b4ad9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6042, \"end_char_idx\": 9124, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "ref_doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "chroma:document": "<PipelineTag pipeline=\"token-classification\"/>\n\n- [`DistilBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).\n- [`TFDistilBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [`FlaxDistilBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`DistilBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFDistilBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxDistilBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`DistilBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFDistilBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxDistilBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: aae41262-a6db-4cd4-acb1-47fee0fa5026
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/distilbert", "title": "DistilBERT", "tokens": 3717, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"aae41262-a6db-4cd4-acb1-47fee0fa5026\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef76daf8-0690-4ee7-b552-3927bdd35fb9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3871d1613d76040ce641a5ddd524847b969eb92bcc98223c068ab18273b4dc80\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e478ee79-8414-438a-a4c2-cf7f70d915d2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3f425c9252d0485347da7d4f43d22b078d33ca16c9506d5bf2e0a7b59e5bd42e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d54e045b-fc30-4e51-909b-425e213820e9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e84852c60a2eca68eeb512935e68150a964d7fe87440157d015075acad70fc01\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7706, \"end_char_idx\": 10744, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "ref_doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "chroma:document": "- [`FlaxDistilBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`DistilBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFDistilBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxDistilBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n- [`DistilBertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFDistilBertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n\u2697\ufe0f Optimization\n\n- A blog post on how to [quantize DistilBERT with \ud83e\udd17 Optimum and Intel](https://huggingface.co/blog/intel).\n- A blog post on how [Optimizing Transformers for GPUs with \ud83e\udd17 Optimum](https://www.philschmid.de/optimizing-transformers-with-optimum-gpu).\n- A blog post on [Optimizing Transformers with Hugging Face Optimum](https://www.philschmid.de/optimizing-transformers-with-optimum).\n\n\u26a1\ufe0f Inference\n\n- A blog post on how to [Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia](https://huggingface.co/blog/bert-inferentia-sagemaker) with DistilBERT.\n- A blog post on [Serverless Inference with Hugging Face's Transformers, DistilBERT and Amazon SageMaker](https://www.philschmid.de/sagemaker-serverless-huggingface-distilbert).\n\n\ud83d\ude80 Deploy\n\n- A blog post on how to [deploy DistilBERT on Google Cloud](https://huggingface.co/blog/how-to-deploy-a-pipeline-to-google-clouds)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d54e045b-fc30-4e51-909b-425e213820e9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/distilbert", "title": "DistilBERT", "tokens": 3717, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d54e045b-fc30-4e51-909b-425e213820e9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef76daf8-0690-4ee7-b552-3927bdd35fb9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3871d1613d76040ce641a5ddd524847b969eb92bcc98223c068ab18273b4dc80\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aae41262-a6db-4cd4-acb1-47fee0fa5026\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"97c94bef943f265e8e484216de7cfe74ffe8bbdf84bfe87ed11b1e9e9c368355\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f2bd44d8-1542-4348-b87e-f9e9f96ce742\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fc654a643be3077f9e5356983828b1b1c5501f6e594087e7e29eb8b3cc07c5c5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9490, \"end_char_idx\": 12157, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "ref_doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "chroma:document": "- [`TFDistilBertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n\u2697\ufe0f Optimization\n\n- A blog post on how to [quantize DistilBERT with \ud83e\udd17 Optimum and Intel](https://huggingface.co/blog/intel).\n- A blog post on how [Optimizing Transformers for GPUs with \ud83e\udd17 Optimum](https://www.philschmid.de/optimizing-transformers-with-optimum-gpu).\n- A blog post on [Optimizing Transformers with Hugging Face Optimum](https://www.philschmid.de/optimizing-transformers-with-optimum).\n\n\u26a1\ufe0f Inference\n\n- A blog post on how to [Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia](https://huggingface.co/blog/bert-inferentia-sagemaker) with DistilBERT.\n- A blog post on [Serverless Inference with Hugging Face's Transformers, DistilBERT and Amazon SageMaker](https://www.philschmid.de/sagemaker-serverless-huggingface-distilbert).\n\n\ud83d\ude80 Deploy\n\n- A blog post on how to [deploy DistilBERT on Google Cloud](https://huggingface.co/blog/how-to-deploy-a-pipeline-to-google-clouds).\n- A blog post on how to [deploy DistilBERT with Amazon SageMaker](https://huggingface.co/blog/deploy-hugging-face-models-easily-with-amazon-sagemaker).\n- A blog post on how to [Deploy BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module](https://www.philschmid.de/terraform-huggingface-amazon-sagemaker).\n\n\n## Combining DistilBERT and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoTokenizer, AutoModel\n\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> tokenizer = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased')\n>>> model = AutoModel.from_pretrained(\"distilbert/distilbert-base-uncased\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n\n>>> text = \"Replace me by any text you'd like.\"\n\n>>> encoded_input = tokenizer(text, return_tensors='pt').to(device)\n>>> model.to(device)\n\n>>> output = model(**encoded_input)\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f2bd44d8-1542-4348-b87e-f9e9f96ce742
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/distilbert", "title": "DistilBERT", "tokens": 3717, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f2bd44d8-1542-4348-b87e-f9e9f96ce742\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ef76daf8-0690-4ee7-b552-3927bdd35fb9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3871d1613d76040ce641a5ddd524847b969eb92bcc98223c068ab18273b4dc80\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d54e045b-fc30-4e51-909b-425e213820e9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/distilbert\", \"title\": \"DistilBERT\", \"tokens\": 3717, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c8c05ce2f7fd2c82997230ff0d6ba86db807b28f2a9188d4af475b3514bec7b4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10603, \"end_char_idx\": 14023, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "ref_doc_id": "ef76daf8-0690-4ee7-b552-3927bdd35fb9", "chroma:document": "\ud83d\ude80 Deploy\n\n- A blog post on how to [deploy DistilBERT on Google Cloud](https://huggingface.co/blog/how-to-deploy-a-pipeline-to-google-clouds).\n- A blog post on how to [deploy DistilBERT with Amazon SageMaker](https://huggingface.co/blog/deploy-hugging-face-models-easily-with-amazon-sagemaker).\n- A blog post on how to [Deploy BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module](https://www.philschmid.de/terraform-huggingface-amazon-sagemaker).\n\n\n## Combining DistilBERT and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoTokenizer, AutoModel\n\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> tokenizer = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased')\n>>> model = AutoModel.from_pretrained(\"distilbert/distilbert-base-uncased\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n\n>>> text = \"Replace me by any text you'd like.\"\n\n>>> encoded_input = tokenizer(text, return_tensors='pt').to(device)\n>>> model.to(device)\n\n>>> output = model(**encoded_input)\n```\n\n\n## DistilBertConfig\n\n[[autodoc]] DistilBertConfig\n\n## DistilBertTokenizer\n\n[[autodoc]] DistilBertTokenizer\n\n## DistilBertTokenizerFast\n\n[[autodoc]] DistilBertTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## DistilBertModel\n\n[[autodoc]] DistilBertModel\n    - forward\n\n## DistilBertForMaskedLM\n\n[[autodoc]] DistilBertForMaskedLM\n    - forward\n\n## DistilBertForSequenceClassification\n\n[[autodoc]] DistilBertForSequenceClassification\n    - forward\n\n## DistilBertForMultipleChoice\n\n[[autodoc]] DistilBertForMultipleChoice\n    - forward\n\n## DistilBertForTokenClassification\n\n[[autodoc]] DistilBertForTokenClassification\n    - forward\n\n## DistilBertForQuestionAnswering\n\n[[autodoc]] DistilBertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFDistilBertModel\n\n[[autodoc]] TFDistilBertModel\n    - call\n\n## TFDistilBertForMaskedLM\n\n[[autodoc]] TFDistilBertForMaskedLM\n    - call\n\n## TFDistilBertForSequenceClassification\n\n[[autodoc]] TFDistilBertForSequenceClassification\n    - call\n\n## TFDistilBertForMultipleChoice\n\n[[autodoc]] TFDistilBertForMultipleChoice\n    - call\n\n## TFDistilBertForTokenClassification\n\n[[autodoc]] TFDistilBertForTokenClassification\n    - call\n\n## TFDistilBertForQuestionAnswering\n\n[[autodoc]] TFDistilBertForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxDistilBertModel\n\n[[autodoc]] FlaxDistilBertModel\n    - __call__\n\n## FlaxDistilBertForMaskedLM\n\n[[autodoc]] FlaxDistilBertForMaskedLM\n    - __call__\n\n## FlaxDistilBertForSequenceClassification\n\n[[autodoc]] FlaxDistilBertForSequenceClassification\n    - __call__\n\n## FlaxDistilBertForMultipleChoice\n\n[[autodoc]] FlaxDistilBertForMultipleChoice\n    - __call__\n\n## FlaxDistilBertForTokenClassification\n\n[[autodoc]] FlaxDistilBertForTokenClassification\n    - __call__\n\n## FlaxDistilBertForQuestionAnswering\n\n[[autodoc]] FlaxDistilBertForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4af69c81-070b-461e-9304-856e6331cace
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/openai-gpt", "title": "OpenAI GPT", "tokens": 2009, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4af69c81-070b-461e-9304-856e6331cace\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87ed7dca-febc-4c1a-aafe-34dba8233b12\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af0e726e3f0aa2f1d1d817bb8d5c0114dadb17d9dbd055bf9d3736932ea8ea91\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d242d551-3876-4bd9-96b3-6f54bbf6f3be\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7392f80b410be9f37e2f03eaa7bc3861ec683e83d40444435df0345d40c48119\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3196, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "doc_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "ref_doc_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "chroma:document": "# OpenAI GPT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=openai-gpt\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-openai--gpt-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/openai-gpt\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nOpenAI GPT model was proposed in [Improving Language Understanding by Generative Pre-Training](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)\nby Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. It's a causal (unidirectional) transformer\npre-trained using language modeling on a large corpus will long range dependencies, the Toronto Book Corpus.\n\nThe abstract from the paper is the following:\n\n*Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering,\nsemantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant,\nlabeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to\nperform adequately. We demonstrate that large gains on these tasks can be realized by generative pretraining of a\nlanguage model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In\ncontrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve\neffective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our\napproach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms\ndiscriminatively trained models that use architectures specifically crafted for each task, significantly improving upon\nthe state of the art in 9 out of the 12 tasks studied.*\n\n[Write With Transformer](https://transformer.huggingface.co/doc/gpt) is a webapp created and hosted by Hugging Face\nshowcasing the generative capabilities of several models. GPT is one of them.\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/openai/finetune-transformer-lm).\n\n## Usage tips\n\n- GPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- GPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next\n  token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be\n  observed in the *run_generation.py* example script.\n\n\nNote:\n\nIf you want to reproduce the original tokenization process of the *OpenAI GPT* paper, you will need to install `ftfy`\nand `SpaCy`:\n\n```bash\npip install spacy ftfy==4.4.3\npython -m spacy download en\n```\n\nIf you don't install `ftfy` and `SpaCy`, the [`OpenAIGPTTokenizer`] will default to tokenize\nusing BERT's `BasicTokenizer` followed by Byte-Pair Encoding (which should be fine for most usage, don't worry)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d242d551-3876-4bd9-96b3-6f54bbf6f3be
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/openai-gpt", "title": "OpenAI GPT", "tokens": 2009, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d242d551-3876-4bd9-96b3-6f54bbf6f3be\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87ed7dca-febc-4c1a-aafe-34dba8233b12\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af0e726e3f0aa2f1d1d817bb8d5c0114dadb17d9dbd055bf9d3736932ea8ea91\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4af69c81-070b-461e-9304-856e6331cace\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a1f0b5cd1ccfbaa4cbae914ceb21c844c3227d215e4f5a9d8ea6d24b6f26e764\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a999f5cd-f2f0-44c2-8f83-fcec7ba76da8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0e0aa7aa444e6a07508276aac7d1d6682f6d8480dc1d0385a4b63b9ded27598e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2781, \"end_char_idx\": 5597, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "doc_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "ref_doc_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "chroma:document": "Note:\n\nIf you want to reproduce the original tokenization process of the *OpenAI GPT* paper, you will need to install `ftfy`\nand `SpaCy`:\n\n```bash\npip install spacy ftfy==4.4.3\npython -m spacy download en\n```\n\nIf you don't install `ftfy` and `SpaCy`, the [`OpenAIGPTTokenizer`] will default to tokenize\nusing BERT's `BasicTokenizer` followed by Byte-Pair Encoding (which should be fine for most usage, don't worry).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with OpenAI GPT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [outperforming OpenAI GPT-3 with SetFit for text-classification](https://www.philschmid.de/getting-started-setfit).\n- See also: [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A blog on how to [Finetune a non-English GPT-2 Model with Hugging Face](https://www.philschmid.de/fine-tune-a-non-english-gpt-2-model-with-huggingface).\n- A blog on [How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate) with GPT-2.\n- A blog on [Training CodeParrot \ud83e\udd9c from Scratch](https://huggingface.co/blog/codeparrot), a large GPT-2 model.\n- A blog on [Faster Text Generation with TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate) with GPT-2.\n- A blog on [How to train a Language Model with Megatron-LM](https://huggingface.co/blog/megatron-training) with a GPT-2 model.\n- A notebook on how to [finetune GPT2 to generate lyrics in the style of your favorite artist](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). \ud83c\udf0e\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`OpenAIGPTLMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-generation/run_generation.py) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a999f5cd-f2f0-44c2-8f83-fcec7ba76da8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/openai-gpt", "title": "OpenAI GPT", "tokens": 2009, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a999f5cd-f2f0-44c2-8f83-fcec7ba76da8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87ed7dca-febc-4c1a-aafe-34dba8233b12\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af0e726e3f0aa2f1d1d817bb8d5c0114dadb17d9dbd055bf9d3736932ea8ea91\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d242d551-3876-4bd9-96b3-6f54bbf6f3be\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3936c892a571caaefb80a73e4d33634a3d4e657637dae0248e9fc759ed063438\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"42c249a1-71a5-4f35-8115-1f78b06ce0bb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5831133e5fd41688d37ffec6e6b9836542dfba4dd92f061344c7072138d94283\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4180, \"end_char_idx\": 6148, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "doc_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "ref_doc_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "chroma:document": "- A blog on [Training CodeParrot \ud83e\udd9c from Scratch](https://huggingface.co/blog/codeparrot), a large GPT-2 model.\n- A blog on [Faster Text Generation with TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate) with GPT-2.\n- A blog on [How to train a Language Model with Megatron-LM](https://huggingface.co/blog/megatron-training) with a GPT-2 model.\n- A notebook on how to [finetune GPT2 to generate lyrics in the style of your favorite artist](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). \ud83c\udf0e\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`OpenAIGPTLMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-generation/run_generation.py) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFOpenAIGPTLMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- See also: [Causal language modeling task guide](../tasks/language_modeling)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A course material on [Byte-Pair Encoding tokenization](https://huggingface.co/course/en/chapter6/5)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 42c249a1-71a5-4f35-8115-1f78b06ce0bb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/openai-gpt", "title": "OpenAI GPT", "tokens": 2009, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"42c249a1-71a5-4f35-8115-1f78b06ce0bb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87ed7dca-febc-4c1a-aafe-34dba8233b12\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af0e726e3f0aa2f1d1d817bb8d5c0114dadb17d9dbd055bf9d3736932ea8ea91\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a999f5cd-f2f0-44c2-8f83-fcec7ba76da8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"09ed59f80e21e44dbb573a5fe5b3978e1cb09f7b07045663de57270509191365\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4743, \"end_char_idx\": 7224, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "doc_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "ref_doc_id": "87ed7dca-febc-4c1a-aafe-34dba8233b12", "chroma:document": "\ud83c\udf0e\n- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). \ud83c\udf0e\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`OpenAIGPTLMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-generation/run_generation.py) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFOpenAIGPTLMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- See also: [Causal language modeling task guide](../tasks/language_modeling)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A course material on [Byte-Pair Encoding tokenization](https://huggingface.co/course/en/chapter6/5).\n\n## OpenAIGPTConfig\n\n[[autodoc]] OpenAIGPTConfig\n\n## OpenAIGPTTokenizer\n\n[[autodoc]] OpenAIGPTTokenizer\n    - save_vocabulary\n\n## OpenAIGPTTokenizerFast\n\n[[autodoc]] OpenAIGPTTokenizerFast\n\n## OpenAI specific outputs\n\n[[autodoc]] models.openai.modeling_openai.OpenAIGPTDoubleHeadsModelOutput\n\n[[autodoc]] models.openai.modeling_tf_openai.TFOpenAIGPTDoubleHeadsModelOutput\n\n<frameworkcontent>\n<pt>\n\n## OpenAIGPTModel\n\n[[autodoc]] OpenAIGPTModel\n    - forward\n\n## OpenAIGPTLMHeadModel\n\n[[autodoc]] OpenAIGPTLMHeadModel\n    - forward\n\n## OpenAIGPTDoubleHeadsModel\n\n[[autodoc]] OpenAIGPTDoubleHeadsModel\n    - forward\n\n## OpenAIGPTForSequenceClassification\n\n[[autodoc]] OpenAIGPTForSequenceClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFOpenAIGPTModel\n\n[[autodoc]] TFOpenAIGPTModel\n    - call\n\n## TFOpenAIGPTLMHeadModel\n\n[[autodoc]] TFOpenAIGPTLMHeadModel\n    - call\n\n## TFOpenAIGPTDoubleHeadsModel\n\n[[autodoc]] TFOpenAIGPTDoubleHeadsModel\n    - call\n\n## TFOpenAIGPTForSequenceClassification\n\n[[autodoc]] TFOpenAIGPTForSequenceClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f7354b6b-35b4-4af4-b8c6-e908da7090af
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/levit", "title": "LeViT", "tokens": 1578, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f7354b6b-35b4-4af4-b8c6-e908da7090af\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab6c5aa6-527b-4c5b-a7c2-73e44a344e59\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08ddfd157b56f771c9af038fc5f787b347fffa193bcb5d305d60af400ea94fff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"61069cfd-3964-47f6-a0e0-123fbe569897\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ed2cce1163764d34d065d8fc141fbfca552170835c3a66fb0f4ef5947a430e93\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab6c5aa6-527b-4c5b-a7c2-73e44a344e59", "doc_id": "ab6c5aa6-527b-4c5b-a7c2-73e44a344e59", "ref_doc_id": "ab6c5aa6-527b-4c5b-a7c2-73e44a344e59", "chroma:document": "# LeViT\n\n## Overview\n\nThe LeViT model was proposed in [LeViT: Introducing Convolutions to Vision Transformers](https://arxiv.org/abs/2104.01136) by Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv\u00e9 J\u00e9gou, Matthijs Douze. LeViT improves the [Vision Transformer (ViT)](vit) in performance and efficiency by a few architectural differences such as activation maps with decreasing resolutions in Transformers and the introduction of an attention bias to integrate positional information.\n\nThe abstract from the paper is the following:\n\n*We design a family of image classification architectures that optimize the trade-off between accuracy\nand efficiency in a high-speed regime. Our work exploits recent findings in attention-based architectures,\nwhich are competitive on highly parallel processing hardware. We revisit principles from the extensive\nliterature on convolutional neural networks to apply them to transformers, in particular activation maps\nwith decreasing resolutions. We also introduce the attention bias, a new way to integrate positional information\nin vision transformers. As a result, we propose LeVIT: a hybrid neural network for fast inference image classification.\nWe consider different measures of efficiency on different hardware platforms, so as to best reflect a wide range of\napplication scenarios. Our extensive experiments empirically validate our technical choices and show they are suitable\nto most architectures. Overall, LeViT significantly outperforms existing convnets and vision transformers with respect\nto the speed/accuracy tradeoff. For example, at 80% ImageNet top-1 accuracy, LeViT is 5 times faster than EfficientNet on CPU. *\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/levit_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> LeViT Architecture. Taken from the <a href=\"https://arxiv.org/abs/2104.01136\">original paper</a>.</small>\n\nThis model was contributed by [anugunj](https://huggingface.co/anugunj). The original code can be found [here](https://github.com/facebookresearch/LeViT).\n\n## Usage tips\n\n- Compared to ViT, LeViT models use an additional distillation head to effectively learn from a teacher (which, in the LeViT paper, is a ResNet like-model). The distillation head is learned through backpropagation under supervision of a ResNet like-model. They also draw inspiration from convolution neural networks to use activation maps with decreasing resolutions to increase the efficiency.\n- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top\n  of the final hidden state and not using the distillation head, or (2) by placing both a prediction head and distillation\n  head on top of the final hidden state. In that case, the prediction head is trained using regular cross-entropy between\n  the prediction of the head and the ground-truth label, while the distillation prediction head is trained using hard distillation\n  (cross-entropy between the prediction of the distillation head and the label predicted by the teacher). At inference time,\n  one takes the average prediction between both heads as final prediction. (2) is also called \"fine-tuning with distillation\",\n  because one relies on a teacher that has already been fine-tuned on the downstream dataset. In terms of models, (1) corresponds\n  to [`LevitForImageClassification`] and (2) corresponds to [`LevitForImageClassificationWithTeacher`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 61069cfd-3964-47f6-a0e0-123fbe569897
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/levit", "title": "LeViT", "tokens": 1578, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"61069cfd-3964-47f6-a0e0-123fbe569897\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab6c5aa6-527b-4c5b-a7c2-73e44a344e59\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08ddfd157b56f771c9af038fc5f787b347fffa193bcb5d305d60af400ea94fff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f7354b6b-35b4-4af4-b8c6-e908da7090af\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a27a930a3a062f7af22c0f8f7a63949c0c4524782925977153ef143e7e90ea81\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"763f5643-3803-485a-bf7e-eabde94095fc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ae41758e61637e18eaaf463a55e41a5fc36f38178757d9252f4c1a0e516b331\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1867, \"end_char_idx\": 4977, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab6c5aa6-527b-4c5b-a7c2-73e44a344e59", "doc_id": "ab6c5aa6-527b-4c5b-a7c2-73e44a344e59", "ref_doc_id": "ab6c5aa6-527b-4c5b-a7c2-73e44a344e59", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2104.01136\">original paper</a>.</small>\n\nThis model was contributed by [anugunj](https://huggingface.co/anugunj). The original code can be found [here](https://github.com/facebookresearch/LeViT).\n\n## Usage tips\n\n- Compared to ViT, LeViT models use an additional distillation head to effectively learn from a teacher (which, in the LeViT paper, is a ResNet like-model). The distillation head is learned through backpropagation under supervision of a ResNet like-model. They also draw inspiration from convolution neural networks to use activation maps with decreasing resolutions to increase the efficiency.\n- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top\n  of the final hidden state and not using the distillation head, or (2) by placing both a prediction head and distillation\n  head on top of the final hidden state. In that case, the prediction head is trained using regular cross-entropy between\n  the prediction of the head and the ground-truth label, while the distillation prediction head is trained using hard distillation\n  (cross-entropy between the prediction of the distillation head and the label predicted by the teacher). At inference time,\n  one takes the average prediction between both heads as final prediction. (2) is also called \"fine-tuning with distillation\",\n  because one relies on a teacher that has already been fine-tuned on the downstream dataset. In terms of models, (1) corresponds\n  to [`LevitForImageClassification`] and (2) corresponds to [`LevitForImageClassificationWithTeacher`].\n- All released checkpoints were pre-trained and fine-tuned on  [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k)\n  (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). only. No external data was used. This is in\n  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for\n  pre-training.\n- The authors of LeViT released 5 trained LeViT models, which you can directly plug into [`LevitModel`] or [`LevitForImageClassification`].\n  Techniques like data augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset\n  (while only using ImageNet-1k for pre-training). The 5 variants available are (all trained on images of size 224x224):\n  *facebook/levit-128S*, *facebook/levit-128*, *facebook/levit-192*, *facebook/levit-256* and\n  *facebook/levit-384*. Note that one should use [`LevitImageProcessor`] in order to\n  prepare images for the model.\n- [`LevitForImageClassificationWithTeacher`] currently supports only inference and not training or fine-tuning.\n- You can check out demo notebooks regarding inference as well as fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer)\n  (you can just replace [`ViTFeatureExtractor`] by [`LevitImageProcessor`] and [`ViTForImageClassification`] by [`LevitForImageClassification`] or [`LevitForImageClassificationWithTeacher`])."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 763f5643-3803-485a-bf7e-eabde94095fc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/levit", "title": "LeViT", "tokens": 1578, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"763f5643-3803-485a-bf7e-eabde94095fc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab6c5aa6-527b-4c5b-a7c2-73e44a344e59\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08ddfd157b56f771c9af038fc5f787b347fffa193bcb5d305d60af400ea94fff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"61069cfd-3964-47f6-a0e0-123fbe569897\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3452ccef3947123c5777c05963c867affa70a7467bc335484171e2de8989693a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3502, \"end_char_idx\": 6188, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab6c5aa6-527b-4c5b-a7c2-73e44a344e59", "doc_id": "ab6c5aa6-527b-4c5b-a7c2-73e44a344e59", "ref_doc_id": "ab6c5aa6-527b-4c5b-a7c2-73e44a344e59", "chroma:document": "- All released checkpoints were pre-trained and fine-tuned on  [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k)\n  (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). only. No external data was used. This is in\n  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for\n  pre-training.\n- The authors of LeViT released 5 trained LeViT models, which you can directly plug into [`LevitModel`] or [`LevitForImageClassification`].\n  Techniques like data augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset\n  (while only using ImageNet-1k for pre-training). The 5 variants available are (all trained on images of size 224x224):\n  *facebook/levit-128S*, *facebook/levit-128*, *facebook/levit-192*, *facebook/levit-256* and\n  *facebook/levit-384*. Note that one should use [`LevitImageProcessor`] in order to\n  prepare images for the model.\n- [`LevitForImageClassificationWithTeacher`] currently supports only inference and not training or fine-tuning.\n- You can check out demo notebooks regarding inference as well as fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer)\n  (you can just replace [`ViTFeatureExtractor`] by [`LevitImageProcessor`] and [`ViTForImageClassification`] by [`LevitForImageClassification`] or [`LevitForImageClassificationWithTeacher`]).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LeViT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`LevitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## LevitConfig\n\n[[autodoc]] LevitConfig\n\n## LevitFeatureExtractor\n\n[[autodoc]] LevitFeatureExtractor\n    - __call__\n\n## LevitImageProcessor\n\n  [[autodoc]] LevitImageProcessor\n    - preprocess\n\n## LevitModel\n\n[[autodoc]] LevitModel\n    - forward\n\n## LevitForImageClassification\n\n[[autodoc]] LevitForImageClassification\n    - forward\n\n## LevitForImageClassificationWithTeacher\n\n[[autodoc]] LevitForImageClassificationWithTeacher\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e8acd7f7-d759-494b-8b5b-9e9e70fcf725
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v2", "title": "MobileNet V2", "tokens": 1526, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e8acd7f7-d759-494b-8b5b-9e9e70fcf725\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"86f676fe-5797-4abb-8b1d-de17bb6258dc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"120e8ad1559fa83bc26316e7c5d0caaaf6fb8b41fcee7af3745aa72828a80c74\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2454bf5a-cd77-46f3-9a08-088d04d3f13f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b86f4f7eaf5a1186616086a2ea84c2a97f70ce75d296891b4cc95fe30193579\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3361, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "86f676fe-5797-4abb-8b1d-de17bb6258dc", "doc_id": "86f676fe-5797-4abb-8b1d-de17bb6258dc", "ref_doc_id": "86f676fe-5797-4abb-8b1d-de17bb6258dc", "chroma:document": "# MobileNet V2\n\n## Overview\n\nThe MobileNet model was proposed in [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://arxiv.org/abs/1801.04381) by Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen.\n\nThe abstract from the paper is the following:\n\n*In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3.*\n\n*The MobileNetV2 architecture is based on an inverted residual structure where the input and output of the residual block are thin bottleneck layers opposite to traditional residual models which use expanded representations in the input an MobileNetV2 uses lightweight depthwise convolutions to filter features in the intermediate expansion layer. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on Imagenet classification, COCO object detection, VOC image segmentation. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as the number of parameters.*\n\nThis model was contributed by [matthijs](https://huggingface.co/Matthijs). The original code and weights can be found [here for the main model](https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet) and [here for DeepLabV3+](https://github.com/tensorflow/models/tree/master/research/deeplab).\n\n## Usage tips\n\n- The checkpoints are named **mobilenet\\_v2\\_*depth*\\_*size***, for example **mobilenet\\_v2\\_1.0\\_224**, where **1.0** is the depth multiplier (sometimes also referred to as \"alpha\" or the width multiplier) and **224** is the resolution of the input images the model was trained on.\n\n- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.\n\n- One can use [`MobileNetV2ImageProcessor`] to prepare images for the model.\n\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra \u201cbackground\u201d class (index 0).\n\n- The segmentation model uses a [DeepLabV3+](https://arxiv.org/abs/1802.02611) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).\n\n- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2454bf5a-cd77-46f3-9a08-088d04d3f13f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v2", "title": "MobileNet V2", "tokens": 1526, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2454bf5a-cd77-46f3-9a08-088d04d3f13f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"86f676fe-5797-4abb-8b1d-de17bb6258dc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"120e8ad1559fa83bc26316e7c5d0caaaf6fb8b41fcee7af3745aa72828a80c74\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e8acd7f7-d759-494b-8b5b-9e9e70fcf725\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ab9d694aa61a4fdea5278caaff1430c7d6f763dfa4e97b69093cc97c02246414\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a37445bf-71b4-43c2-a466-ddc443e60125\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9638d3cc2e017925b8999b907b772b33b30a82e487d7b83f92dced7945289c87\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1849, \"end_char_idx\": 5060, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "86f676fe-5797-4abb-8b1d-de17bb6258dc", "doc_id": "86f676fe-5797-4abb-8b1d-de17bb6258dc", "ref_doc_id": "86f676fe-5797-4abb-8b1d-de17bb6258dc", "chroma:document": "The original code and weights can be found [here for the main model](https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet) and [here for DeepLabV3+](https://github.com/tensorflow/models/tree/master/research/deeplab).\n\n## Usage tips\n\n- The checkpoints are named **mobilenet\\_v2\\_*depth*\\_*size***, for example **mobilenet\\_v2\\_1.0\\_224**, where **1.0** is the depth multiplier (sometimes also referred to as \"alpha\" or the width multiplier) and **224** is the resolution of the input images the model was trained on.\n\n- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.\n\n- One can use [`MobileNetV2ImageProcessor`] to prepare images for the model.\n\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra \u201cbackground\u201d class (index 0).\n\n- The segmentation model uses a [DeepLabV3+](https://arxiv.org/abs/1802.02611) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).\n\n- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size. To use native PyTorch padding behavior, create a [`MobileNetV2Config`] with `tf_padding = False`.\n\nUnsupported features:\n\n- The [`MobileNetV2Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use an average pooling layer with a fixed 7x7 window and stride 1 instead of global pooling. For inputs that are larger than the recommended image size, this gives a pooled output that is larger than 1x1. The Hugging Face implementation does not support this.\n\n- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional \"FakeQuantization\" operations to unquantize the weights.\n\n- It's common to extract the output from the expansion layers at indices 10 and 13, as well as the output from the final 1x1 convolution layer, for downstream purposes. Using `output_hidden_states=True` returns the output from all intermediate layers. There is currently no way to limit this to specific layers.\n\n- The DeepLabV3+ segmentation head does not use the final convolution layer from the backbone, but this layer gets computed anyway. There is currently no way to tell [`MobileNetV2Model`] up to which layer it should run.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with MobileNetV2.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`MobileNetV2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a37445bf-71b4-43c2-a466-ddc443e60125
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v2", "title": "MobileNet V2", "tokens": 1526, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a37445bf-71b4-43c2-a466-ddc443e60125\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"86f676fe-5797-4abb-8b1d-de17bb6258dc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"120e8ad1559fa83bc26316e7c5d0caaaf6fb8b41fcee7af3745aa72828a80c74\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2454bf5a-cd77-46f3-9a08-088d04d3f13f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e63799d2b8ae919d6b10295b27a2b49f42cb15a54f3f62df378905f6a6a5196c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3362, \"end_char_idx\": 6037, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "86f676fe-5797-4abb-8b1d-de17bb6258dc", "doc_id": "86f676fe-5797-4abb-8b1d-de17bb6258dc", "ref_doc_id": "86f676fe-5797-4abb-8b1d-de17bb6258dc", "chroma:document": "To use native PyTorch padding behavior, create a [`MobileNetV2Config`] with `tf_padding = False`.\n\nUnsupported features:\n\n- The [`MobileNetV2Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use an average pooling layer with a fixed 7x7 window and stride 1 instead of global pooling. For inputs that are larger than the recommended image size, this gives a pooled output that is larger than 1x1. The Hugging Face implementation does not support this.\n\n- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional \"FakeQuantization\" operations to unquantize the weights.\n\n- It's common to extract the output from the expansion layers at indices 10 and 13, as well as the output from the final 1x1 convolution layer, for downstream purposes. Using `output_hidden_states=True` returns the output from all intermediate layers. There is currently no way to limit this to specific layers.\n\n- The DeepLabV3+ segmentation head does not use the final convolution layer from the backbone, but this layer gets computed anyway. There is currently no way to tell [`MobileNetV2Model`] up to which layer it should run.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with MobileNetV2.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`MobileNetV2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n**Semantic segmentation**\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## MobileNetV2Config\n\n[[autodoc]] MobileNetV2Config\n\n## MobileNetV2FeatureExtractor\n\n[[autodoc]] MobileNetV2FeatureExtractor\n    - preprocess\n    - post_process_semantic_segmentation\n\n## MobileNetV2ImageProcessor\n\n[[autodoc]] MobileNetV2ImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n## MobileNetV2Model\n\n[[autodoc]] MobileNetV2Model\n    - forward\n\n## MobileNetV2ForImageClassification\n\n[[autodoc]] MobileNetV2ForImageClassification\n    - forward\n\n## MobileNetV2ForSemanticSegmentation\n\n[[autodoc]] MobileNetV2ForSemanticSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4937576a-689c-491f-9d30-1fad70396673
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4937576a-689c-491f-9d30-1fad70396673\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fd4a1280-1450-4452-8793-d8a0b87f234e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"722ebb0a-60bf-4a31-b4d9-8b8c18d86eb9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aa77a9a75339f6f23f8045526df80b39976f8ffa151274d52e2e3ab690e7c9bb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3782, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "ref_doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "chroma:document": "# MusicGen Melody\n\n## Overview\n\nThe MusicGen Melody model was proposed in [Simple and Controllable Music Generation](https://arxiv.org/abs/2306.05284) by Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi and Alexandre D\u00e9fossez.\n\nMusicGen Melody is a single stage auto-regressive Transformer model capable of generating high-quality music samples conditioned on text descriptions or audio prompts. The text descriptions are passed through a frozen text encoder model to obtain a sequence of hidden-state representations. MusicGen is then trained to predict discrete audio tokens, or *audio codes*, conditioned on these hidden-states. These audio tokens are then decoded using an audio compression model, such as EnCodec, to recover the audio waveform.\n\nThrough an efficient token interleaving pattern, MusicGen does not require a self-supervised semantic representation of the text/audio prompts, thus eliminating the need to cascade multiple models to predict a set of codebooks (e.g. hierarchically or upsampling). Instead, it is able to generate all the codebooks in a single forward pass.\n\nThe abstract from the paper is the following:\n\n*We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen.*\n\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/audiocraft). The pre-trained checkpoints can be found on the [Hugging Face Hub](https://huggingface.co/models?sort=downloads&search=facebook%2Fmusicgen).\n\n\n## Difference with [MusicGen](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen)\n\nThere are two key differences with MusicGen:\n1. The audio prompt is used here as a conditional signal for the generated audio sample, whereas it's used for audio continuation in [MusicGen](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen).\n2. Conditional text and audio signals are concatenated to the decoder's hidden states instead of being used as a cross-attention signal, as in MusicGen.\n\n## Generation\n\nMusicGen Melody is compatible with two generation modes: greedy and sampling. In practice, sampling leads to significantly better results than greedy, thus we encourage sampling mode to be used where possible. Sampling is enabled by default, and can be explicitly specified by setting `do_sample=True` in the call to [`MusicgenMelodyForConditionalGeneration.generate`], or by overriding the model's generation config (see below).\n\nTransformers supports both mono (1-channel) and stereo (2-channel) variants of MusicGen Melody. The mono channel versions generate a single set of codebooks. The stereo versions generate 2 sets of codebooks, 1 for each channel (left/right), and each set of codebooks is decoded independently through the audio compression model. The audio streams for each channel are combined to give the final stereo output."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 722ebb0a-60bf-4a31-b4d9-8b8c18d86eb9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"722ebb0a-60bf-4a31-b4d9-8b8c18d86eb9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fd4a1280-1450-4452-8793-d8a0b87f234e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4937576a-689c-491f-9d30-1fad70396673\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42a83f0f168f9130fb3cf8dbe92d4227141b300b4253de2bd116d022d17a3b6b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0b3b41b-5e7a-4e83-ae50-96cbacd3eab2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"96c59f2b0ab90fbbff7f02393fa8c59e88b45ab5f4f913fed51f83acf59317b7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2109, \"end_char_idx\": 5012, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "ref_doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "chroma:document": "This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/audiocraft). The pre-trained checkpoints can be found on the [Hugging Face Hub](https://huggingface.co/models?sort=downloads&search=facebook%2Fmusicgen).\n\n\n## Difference with [MusicGen](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen)\n\nThere are two key differences with MusicGen:\n1. The audio prompt is used here as a conditional signal for the generated audio sample, whereas it's used for audio continuation in [MusicGen](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen).\n2. Conditional text and audio signals are concatenated to the decoder's hidden states instead of being used as a cross-attention signal, as in MusicGen.\n\n## Generation\n\nMusicGen Melody is compatible with two generation modes: greedy and sampling. In practice, sampling leads to significantly better results than greedy, thus we encourage sampling mode to be used where possible. Sampling is enabled by default, and can be explicitly specified by setting `do_sample=True` in the call to [`MusicgenMelodyForConditionalGeneration.generate`], or by overriding the model's generation config (see below).\n\nTransformers supports both mono (1-channel) and stereo (2-channel) variants of MusicGen Melody. The mono channel versions generate a single set of codebooks. The stereo versions generate 2 sets of codebooks, 1 for each channel (left/right), and each set of codebooks is decoded independently through the audio compression model. The audio streams for each channel are combined to give the final stereo output.\n\n\n#### Audio Conditional Generation\n\nThe model can generate an audio sample conditioned on a text and an audio prompt through use of the [`MusicgenMelodyProcessor`] to pre-process the inputs.\n\nIn the following examples, we load an audio file using the \ud83e\udd17 Datasets library, which can be pip installed through the command below:\n\n```\npip install --upgrade pip\npip install datasets[audio]\n```\n\nThe audio file we are about to use is loaded as follows:\n```python\n>>> from datasets import load_dataset\n\n>>> dataset = load_dataset(\"sanchit-gandhi/gtzan\", split=\"train\", streaming=True)\n>>> sample = next(iter(dataset))[\"audio\"]\n```\n\nThe audio prompt should ideally be free of the low-frequency signals usually produced by instruments such as drums and bass. The [Demucs](https://github.com/adefossez/demucs/tree/main) model can be used to separate vocals and other signals from the drums and bass components.\n\nIf you wish to use Demucs, you first need to follow the installation steps [here](https://github.com/adefossez/demucs/tree/main?tab=readme-ov-file#for-musicians) before using the following snippet:\n\n```python\nfrom demucs import pretrained\nfrom demucs.apply import apply_model\nfrom demucs.audio import convert_audio\nimport torch"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e0b3b41b-5e7a-4e83-ae50-96cbacd3eab2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e0b3b41b-5e7a-4e83-ae50-96cbacd3eab2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fd4a1280-1450-4452-8793-d8a0b87f234e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"722ebb0a-60bf-4a31-b4d9-8b8c18d86eb9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af00afebb2b930cdf019f4958c9dc529dbdc4c3762e91e1a1785abde0b679345\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ef434389-dc60-4c5e-b98e-48f1fad4dda2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"440784dd95b26be44a7ceb91f0645fdba070d2a64a7c45b35d9fa45b42d28399\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3785, \"end_char_idx\": 7205, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "ref_doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "chroma:document": "#### Audio Conditional Generation\n\nThe model can generate an audio sample conditioned on a text and an audio prompt through use of the [`MusicgenMelodyProcessor`] to pre-process the inputs.\n\nIn the following examples, we load an audio file using the \ud83e\udd17 Datasets library, which can be pip installed through the command below:\n\n```\npip install --upgrade pip\npip install datasets[audio]\n```\n\nThe audio file we are about to use is loaded as follows:\n```python\n>>> from datasets import load_dataset\n\n>>> dataset = load_dataset(\"sanchit-gandhi/gtzan\", split=\"train\", streaming=True)\n>>> sample = next(iter(dataset))[\"audio\"]\n```\n\nThe audio prompt should ideally be free of the low-frequency signals usually produced by instruments such as drums and bass. The [Demucs](https://github.com/adefossez/demucs/tree/main) model can be used to separate vocals and other signals from the drums and bass components.\n\nIf you wish to use Demucs, you first need to follow the installation steps [here](https://github.com/adefossez/demucs/tree/main?tab=readme-ov-file#for-musicians) before using the following snippet:\n\n```python\nfrom demucs import pretrained\nfrom demucs.apply import apply_model\nfrom demucs.audio import convert_audio\nimport torch\n\n\nwav = torch.tensor(sample[\"array\"]).to(torch.float32)\n\ndemucs = pretrained.get_model('htdemucs')\n\nwav = convert_audio(wav[None], sample[\"sampling_rate\"], demucs.samplerate, demucs.audio_channels)\nwav = apply_model(demucs, wav[None])\n```\n\nYou can then use the following snippet to generate music:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> inputs = processor(\n...     audio=wav,\n...     sampling_rate=demucs.samplerate,\n...     text=[\"80s blues track with groovy saxophone\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nYou can also pass the audio signal directly without using Demucs, although the quality of the generation will probably be degraded:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> inputs = processor(\n...     audio=sample[\"array\"],\n...     sampling_rate=sample[\"sampling_rate\"],\n...     text=[\"80s blues track with groovy saxophone\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe audio outputs are a three-dimensional Torch tensor of shape `(batch_size, num_channels, sequence_length)`. To listen to the generated audio samples, you can either play them in an ipynb notebook:\n\n```python\nfrom IPython.display import Audio\n\nsampling_rate = model.config.audio_encoder.sampling_rate\nAudio(audio_values[0].numpy(), rate=sampling_rate)\n```\n\nOr save them as a `.wav` file using a third-party library, e.g. `soundfile`:\n\n```python\n>>> import soundfile as sf\n\n>>> sampling_rate = model.config.audio_encoder.sampling_rate\n>>> sf.write(\"musicgen_out.wav\", audio_values[0].T.numpy(), sampling_rate)\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ef434389-dc60-4c5e-b98e-48f1fad4dda2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ef434389-dc60-4c5e-b98e-48f1fad4dda2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fd4a1280-1450-4452-8793-d8a0b87f234e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0b3b41b-5e7a-4e83-ae50-96cbacd3eab2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5a88953e4b7307aea1277c995ab4b4d5f9cbd2bc5451ddb70280a2c516e252f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f41bb20c-35cc-4cdb-8bbb-b471723713dd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"da2379d28eab79230bbbf25d14b76d052dc4a5a4986a3642d75cc65e6d96d3d7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7208, \"end_char_idx\": 8402, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "ref_doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "chroma:document": "### Text-only Conditional Generation\n\nThe same [`MusicgenMelodyProcessor`] can be used to pre-process a text-only prompt. \n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> inputs = processor(\n...     text=[\"80s pop track with bassy drums and synth\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe `guidance_scale` is used in classifier free guidance (CFG), setting the weighting between the conditional logits (which are predicted from the text prompts) and the unconditional logits (which are predicted from an unconditional or 'null' prompt). Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer audio quality. CFG is enabled by setting `guidance_scale > 1`. For best results, use `guidance_scale=3` (default)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f41bb20c-35cc-4cdb-8bbb-b471723713dd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f41bb20c-35cc-4cdb-8bbb-b471723713dd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fd4a1280-1450-4452-8793-d8a0b87f234e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ef434389-dc60-4c5e-b98e-48f1fad4dda2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"513117411da4754f01ca70df26fe77488f84411a4e654bea8a61a03a7811e5a8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7922c625-2929-4769-a7c1-0c1dc3fd706d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f92c813daeb7333dc642c71130faecadc99b02fc3032cdb16a376b9506130403\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7208, \"end_char_idx\": 10846, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "ref_doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "chroma:document": "### Text-only Conditional Generation\n\nThe same [`MusicgenMelodyProcessor`] can be used to pre-process a text-only prompt. \n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> inputs = processor(\n...     text=[\"80s pop track with bassy drums and synth\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe `guidance_scale` is used in classifier free guidance (CFG), setting the weighting between the conditional logits (which are predicted from the text prompts) and the unconditional logits (which are predicted from an unconditional or 'null' prompt). Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer audio quality. CFG is enabled by setting `guidance_scale > 1`. For best results, use `guidance_scale=3` (default).\n\n\nYou can also generate in batch:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> # take the first quarter of the audio sample\n>>> sample_1 = sample[\"array\"][: len(sample[\"array\"]) // 4]\n\n>>> # take the first half of the audio sample\n>>> sample_2 = sample[\"array\"][: len(sample[\"array\"]) // 2]\n\n>>> inputs = processor(\n...     audio=[sample_1, sample_2],\n...     sampling_rate=sample[\"sampling_rate\"],\n...     text=[\"80s blues track with groovy saxophone\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\n### Unconditional Generation\n\nThe inputs for unconditional (or 'null') generation can be obtained through the method [`MusicgenMelodyProcessor.get_unconditional_inputs`]:\n\n```python\n>>> from transformers import MusicgenMelodyForConditionalGeneration, MusicgenMelodyProcessor\n\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n>>> unconditional_inputs = MusicgenMelodyProcessor.from_pretrained(\"facebook/musicgen-melody\").get_unconditional_inputs(num_samples=1)\n\n>>> audio_values = model.generate(**unconditional_inputs, do_sample=True, max_new_tokens=256)\n```\n\n### Generation Configuration\n\nThe default parameters that control the generation process, such as sampling, guidance scale and number of generated tokens, can be found in the model's generation config, and updated as desired:\n\n```python\n>>> from transformers import MusicgenMelodyForConditionalGeneration\n\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> # inspect the default generation config\n>>> model.generation_config\n\n>>> # increase the guidance scale to 4.0\n>>> model.generation_config.guidance_scale = 4.0\n\n>>> # decrease the max length to 256 tokens\n>>> model.generation_config.max_length = 256\n```\n\nNote that any arguments passed to the generate method will **supersede** those in the generation config, so setting `do_sample=False` in the call to generate will supersede the setting of `model.generation_config.do_sample` in the generation config."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7922c625-2929-4769-a7c1-0c1dc3fd706d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7922c625-2929-4769-a7c1-0c1dc3fd706d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fd4a1280-1450-4452-8793-d8a0b87f234e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f41bb20c-35cc-4cdb-8bbb-b471723713dd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8be2275e43c5870899c52108cdfbf69735123ac3b1459fe4fec75ed4be5cdb41\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59d22861-d3dd-467a-873b-bfcdc35c4bda\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"89f3b22875cae9e05aa58fe5c5a406219922e37b7353f542321b63cfc2ecd17c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10848, \"end_char_idx\": 13660, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "ref_doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "chroma:document": "## Model Structure\n\nThe MusicGen model can be de-composed into three distinct stages:\n1. Text encoder: maps the text inputs to a sequence of hidden-state representations. The pre-trained MusicGen models use a frozen text encoder from either T5 or Flan-T5.\n2. MusicGen Melody decoder: a language model (LM) that auto-regressively generates audio tokens (or codes) conditional on the encoder hidden-state representations\n3. Audio decoder: used to recover the audio waveform from the audio tokens predicted by the decoder.\n\nThus, the MusicGen model can either be used as a standalone decoder model, corresponding to the class [`MusicgenMelodyForCausalLM`], or as a composite model that includes the text encoder and audio encoder, corresponding to the class [`MusicgenMelodyForConditionalGeneration`]. If only the decoder needs to be loaded from the pre-trained checkpoint, it can be loaded by first specifying the correct config, or be accessed through the `.decoder` attribute of the composite model:\n\n```python\n>>> from transformers import AutoConfig, MusicgenMelodyForCausalLM, MusicgenMelodyForConditionalGeneration\n\n>>> # Option 1: get decoder config and pass to `.from_pretrained`\n>>> decoder_config = AutoConfig.from_pretrained(\"facebook/musicgen-melody\").decoder\n>>> decoder = MusicgenMelodyForCausalLM.from_pretrained(\"facebook/musicgen-melody\", **decoder_config.to_dict())\n\n>>> # Option 2: load the entire composite model, but only return the decoder\n>>> decoder = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\").decoder\n```\n\nSince the text encoder and audio encoder models are frozen during training, the MusicGen decoder [`MusicgenMelodyForCausalLM`] can be trained standalone on a dataset of encoder hidden-states and audio codes. For inference, the trained decoder can be combined with the frozen text encoder and audio encoder to recover the composite [`MusicgenMelodyForConditionalGeneration`] model.\n\n## Checkpoint Conversion\n\n- After downloading the original checkpoints from [here](https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md#importing--exporting-models), you can convert them using the **conversion script** available at `src/transformers/models/musicgen_melody/convert_musicgen_melody_transformers.py` with the following command:\n\n```bash\npython src/transformers/models/musicgen_melody/convert_musicgen_melody_transformers.py \\\n    --checkpoint=\"facebook/musicgen-melody\" --pytorch_dump_folder /output/path \n```\n\nTips:\n* MusicGen is trained on the 32kHz checkpoint of Encodec. You should ensure you use a compatible version of the Encodec model.\n* Sampling mode tends to deliver better results than greedy - you can toggle sampling with the variable `do_sample` in the call to [`MusicgenMelodyForConditionalGeneration.generate`]"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 59d22861-d3dd-467a-873b-bfcdc35c4bda
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"59d22861-d3dd-467a-873b-bfcdc35c4bda\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fd4a1280-1450-4452-8793-d8a0b87f234e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7922c625-2929-4769-a7c1-0c1dc3fd706d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cdfb69c847ea5ddb92c9605be10b500094f8b4ed4611a39ce1ce8f06d7d2d4d7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12625, \"end_char_idx\": 14259, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "ref_doc_id": "fd4a1280-1450-4452-8793-d8a0b87f234e", "chroma:document": "For inference, the trained decoder can be combined with the frozen text encoder and audio encoder to recover the composite [`MusicgenMelodyForConditionalGeneration`] model.\n\n## Checkpoint Conversion\n\n- After downloading the original checkpoints from [here](https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md#importing--exporting-models), you can convert them using the **conversion script** available at `src/transformers/models/musicgen_melody/convert_musicgen_melody_transformers.py` with the following command:\n\n```bash\npython src/transformers/models/musicgen_melody/convert_musicgen_melody_transformers.py \\\n    --checkpoint=\"facebook/musicgen-melody\" --pytorch_dump_folder /output/path \n```\n\nTips:\n* MusicGen is trained on the 32kHz checkpoint of Encodec. You should ensure you use a compatible version of the Encodec model.\n* Sampling mode tends to deliver better results than greedy - you can toggle sampling with the variable `do_sample` in the call to [`MusicgenMelodyForConditionalGeneration.generate`]\n\n\n## MusicgenMelodyDecoderConfig\n\n[[autodoc]] MusicgenMelodyDecoderConfig\n\n## MusicgenMelodyProcessor\n\n[[autodoc]] MusicgenMelodyProcessor\n    - get_unconditional_inputs\n\n## MusicgenMelodyFeatureExtractor\n\n[[autodoc]] MusicgenMelodyFeatureExtractor\n    - _extract_stem_indices\n\n## MusicgenMelodyConfig\n\n[[autodoc]] MusicgenMelodyConfig\n\n## MusicgenMelodyModel\n\n[[autodoc]] MusicgenMelodyModel\n    - forward\n\n## MusicgenMelodyForCausalLM\n\n[[autodoc]] MusicgenMelodyForCausalLM\n    - forward\n\n## MusicgenMelodyForConditionalGeneration\n\n[[autodoc]] MusicgenMelodyForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3ba6ac93-9490-4336-89b3-293c86c93ed6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3ba6ac93-9490-4336-89b3-293c86c93ed6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9446930d-9a3a-4d88-8280-5af6da60f4fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c05921a7-bf19-47e4-b954-5d619875f0ff\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d0538b76f38fe4dfddf7af3012bf9511e9aaad69a357a965501d703028a429bf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2925, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "ref_doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "chroma:document": "# GPT-J\n\n## Overview\n\nThe GPT-J model was released in the [kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax) repository by Ben Wang and Aran Komatsuzaki. It is a GPT-2-like\ncausal language model trained on [the Pile](https://pile.eleuther.ai/) dataset.\n\nThis model was contributed by [Stella Biderman](https://huggingface.co/stellaathena).\n\n## Usage tips\n\n- To load [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) in float32 one would need at least 2x model size\n  RAM: 1x for initial weights and another 1x to load the checkpoint. So for GPT-J it would take at least 48GB\n  RAM to just load the model. To reduce the RAM usage there are a few options. The `torch_dtype` argument can be\n  used to initialize the model in half-precision on a CUDA device only. There is also a fp16 branch which stores the fp16 weights,\n  which could be used to further minimize the RAM usage:\n\n```python\n>>> from transformers import GPTJForCausalLM\n>>> import torch\n\n>>> device = \"cuda\"\n>>> model = GPTJForCausalLM.from_pretrained(\n...     \"EleutherAI/gpt-j-6B\",\n...     revision=\"float16\",\n...     torch_dtype=torch.float16,\n... ).to(device)\n```\n\n- The model should fit on 16GB GPU for inference. For training/fine-tuning it would take much more GPU RAM. Adam\n  optimizer for example makes four copies of the model: model, gradients, average and squared average of the gradients.\n  So it would need at least 4x model size GPU memory, even with mixed precision as gradient updates are in fp32. This\n  is not including the activations and data batches, which would again require some more GPU RAM. So one should explore\n  solutions such as DeepSpeed, to train/fine-tune the model. Another option is to use the original codebase to\n  train/fine-tune the model on TPU and then convert the model to Transformers format for inference. Instructions for\n  that could be found [here](https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md)\n\n- Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer. These extra\n  tokens are added for the sake of efficiency on TPUs. To avoid the mismatch between embedding matrix size and vocab\n  size, the tokenizer for [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) contains 143 extra tokens\n  `<|extratoken_1|>... <|extratoken_143|>`, so the `vocab_size` of tokenizer also becomes 50400.\n\n## Usage examples\n\nThe [`~generation.GenerationMixin.generate`] method can be used to generate text using GPT-J\nmodel.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"EleutherAI/gpt-j-6B\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-j-6B\")\n\n>>> prompt = (\n...     \"In a shocking finding, scientists discovered a herd of unicorns living in a remote, \"\n...     \"previously unexplored valley, in the Andes Mountains."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c05921a7-bf19-47e4-b954-5d619875f0ff
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c05921a7-bf19-47e4-b954-5d619875f0ff\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9446930d-9a3a-4d88-8280-5af6da60f4fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3ba6ac93-9490-4336-89b3-293c86c93ed6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"60a91bc8fe85bb5bfd2ff8b71fdacac99749d46aa2afd985fdd3e38a30dbe600\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2daf4004-cfd5-4db6-aab2-04c4205087cd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8f966e3d8550a2dfc7fb4ce6261a12671387d9e6cd7d5fc7c908a6d7e4bd298a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1403, \"end_char_idx\": 4507, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "ref_doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "chroma:document": "So it would need at least 4x model size GPU memory, even with mixed precision as gradient updates are in fp32. This\n  is not including the activations and data batches, which would again require some more GPU RAM. So one should explore\n  solutions such as DeepSpeed, to train/fine-tune the model. Another option is to use the original codebase to\n  train/fine-tune the model on TPU and then convert the model to Transformers format for inference. Instructions for\n  that could be found [here](https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md)\n\n- Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer. These extra\n  tokens are added for the sake of efficiency on TPUs. To avoid the mismatch between embedding matrix size and vocab\n  size, the tokenizer for [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) contains 143 extra tokens\n  `<|extratoken_1|>... <|extratoken_143|>`, so the `vocab_size` of tokenizer also becomes 50400.\n\n## Usage examples\n\nThe [`~generation.GenerationMixin.generate`] method can be used to generate text using GPT-J\nmodel.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"EleutherAI/gpt-j-6B\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-j-6B\")\n\n>>> prompt = (\n...     \"In a shocking finding, scientists discovered a herd of unicorns living in a remote, \"\n...     \"previously unexplored valley, in the Andes Mountains. Even more surprising to the \"\n...     \"researchers was the fact that the unicorns spoke perfect English.\"\n... )\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n...or in float16 precision:\n\n```python\n>>> from transformers import GPTJForCausalLM, AutoTokenizer\n>>> import torch\n\n>>> device = \"cuda\"\n>>> model = GPTJForCausalLM.from_pretrained(\"EleutherAI/gpt-j-6B\", torch_dtype=torch.float16).to(device)\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-j-6B\")\n\n>>> prompt = (\n...     \"In a shocking finding, scientists discovered a herd of unicorns living in a remote, \"\n...     \"previously unexplored valley, in the Andes Mountains. Even more surprising to the \"\n...     \"researchers was the fact that the unicorns spoke perfect English.\"\n... )\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids.to(device)\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GPT-J. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2daf4004-cfd5-4db6-aab2-04c4205087cd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2daf4004-cfd5-4db6-aab2-04c4205087cd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9446930d-9a3a-4d88-8280-5af6da60f4fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c05921a7-bf19-47e4-b954-5d619875f0ff\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2fccd3de812277d4c85c6e350da6768f83c3612e59d48f12547ebdf1466beffb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"00e51421-d5d3-4357-853c-d5e586eabe8c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bcf573acdfbda20cc3d3718cf1223ecc24decc6444988147c5394faa13b4e153\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2926, \"end_char_idx\": 5529, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "ref_doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "chroma:document": "Even more surprising to the \"\n...     \"researchers was the fact that the unicorns spoke perfect English.\"\n... )\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n...or in float16 precision:\n\n```python\n>>> from transformers import GPTJForCausalLM, AutoTokenizer\n>>> import torch\n\n>>> device = \"cuda\"\n>>> model = GPTJForCausalLM.from_pretrained(\"EleutherAI/gpt-j-6B\", torch_dtype=torch.float16).to(device)\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-j-6B\")\n\n>>> prompt = (\n...     \"In a shocking finding, scientists discovered a herd of unicorns living in a remote, \"\n...     \"previously unexplored valley, in the Andes Mountains. Even more surprising to the \"\n...     \"researchers was the fact that the unicorns spoke perfect English.\"\n... )\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids.to(device)\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GPT-J. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- Description of [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B).\n- A blog on how to [Deploy GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker](https://huggingface.co/blog/gptj-sagemaker).\n- A blog on how to [Accelerate GPT-J inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/gptj-deepspeed-inference).\n- A blog post introducing [GPT-J-6B: 6B JAX-Based Transformer](https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/). \ud83c\udf0e\n- A notebook for [GPT-J-6B Inference Demo](https://colab.research.google.com/github/kingoflolz/mesh-transformer-jax/blob/master/colab_demo.ipynb). \ud83c\udf0e\n- Another notebook demonstrating [Inference with GPT-J-6B](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/GPT-J-6B/Inference_with_GPT_J_6B.ipynb).  \n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 00e51421-d5d3-4357-853c-d5e586eabe8c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"00e51421-d5d3-4357-853c-d5e586eabe8c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9446930d-9a3a-4d88-8280-5af6da60f4fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2daf4004-cfd5-4db6-aab2-04c4205087cd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"193baca8903a9c58ec827b82b75ba95655021ebe97b161d5bb3b2933342799c5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3709facb-f696-4731-bc55-340cbd3b4db2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dee81cc5176a66e797e12929df6d4914d2a0edb93de5a43564f76dd7477630c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4281, \"end_char_idx\": 6640, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "ref_doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "chroma:document": "If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- Description of [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B).\n- A blog on how to [Deploy GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker](https://huggingface.co/blog/gptj-sagemaker).\n- A blog on how to [Accelerate GPT-J inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/gptj-deepspeed-inference).\n- A blog post introducing [GPT-J-6B: 6B JAX-Based Transformer](https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/). \ud83c\udf0e\n- A notebook for [GPT-J-6B Inference Demo](https://colab.research.google.com/github/kingoflolz/mesh-transformer-jax/blob/master/colab_demo.ipynb). \ud83c\udf0e\n- Another notebook demonstrating [Inference with GPT-J-6B](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/GPT-J-6B/Inference_with_GPT_J_6B.ipynb).  \n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`GPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3709facb-f696-4731-bc55-340cbd3b4db2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3709facb-f696-4731-bc55-340cbd3b4db2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9446930d-9a3a-4d88-8280-5af6da60f4fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"00e51421-d5d3-4357-853c-d5e586eabe8c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87f15ef52b27c83968d5568b6f8cdeeb9c793a1337950874527d88a2d68f27d6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5175, \"end_char_idx\": 7703, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "ref_doc_id": "9446930d-9a3a-4d88-8280-5af6da60f4fb", "chroma:document": "\ud83c\udf0e\n- Another notebook demonstrating [Inference with GPT-J-6B](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/GPT-J-6B/Inference_with_GPT_J_6B.ipynb).  \n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`GPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb).\n\n**Documentation resources**\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## GPTJConfig\n\n[[autodoc]] GPTJConfig\n    - all\n\n<frameworkcontent>\n<pt>\n\n## GPTJModel\n\n[[autodoc]] GPTJModel\n    - forward\n\n## GPTJForCausalLM\n\n[[autodoc]] GPTJForCausalLM\n    - forward\n\n## GPTJForSequenceClassification\n\n[[autodoc]] GPTJForSequenceClassification\n    - forward\n\n## GPTJForQuestionAnswering\n\n[[autodoc]] GPTJForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFGPTJModel\n\n[[autodoc]] TFGPTJModel\n    - call\n\n## TFGPTJForCausalLM\n\n[[autodoc]] TFGPTJForCausalLM\n    - call\n\n## TFGPTJForSequenceClassification\n\n[[autodoc]] TFGPTJForSequenceClassification\n    - call\n\n## TFGPTJForQuestionAnswering\n\n[[autodoc]] TFGPTJForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxGPTJModel\n\n[[autodoc]] FlaxGPTJModel\n    - __call__\n\n## FlaxGPTJForCausalLM\n\n[[autodoc]] FlaxGPTJForCausalLM\n    - __call__\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5a5b2dee-5552-4924-9191-4fed639146e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilevit", "title": "MobileViT", "tokens": 1539, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5a5b2dee-5552-4924-9191-4fed639146e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a5aed73d-b889-4ae8-902f-4943066d3c7e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2a6a7626dcd1e35b5340b766066bbe0dbfa17288134bebbef7f2179372837e8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b2638ff-cead-4eb5-b0b5-ca3eea20fdd8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"411cb7705587e026ee8ac6b10a7d1ab7f90c8a09f5a22da18b44060d22afc4f1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3363, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a5aed73d-b889-4ae8-902f-4943066d3c7e", "doc_id": "a5aed73d-b889-4ae8-902f-4943066d3c7e", "ref_doc_id": "a5aed73d-b889-4ae8-902f-4943066d3c7e", "chroma:document": "# MobileViT\n\n## Overview\n\nThe MobileViT model was proposed in [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://arxiv.org/abs/2110.02178) by Sachin Mehta and Mohammad Rastegari. MobileViT introduces a new layer that replaces local processing in convolutions with global processing using transformers.\n\nThe abstract from the paper is the following:\n\n*Light-weight convolutional neural networks (CNNs) are the de-facto for mobile vision tasks. Their spatial inductive biases allow them to learn representations with fewer parameters across different vision tasks. However, these networks are spatially local. To learn global representations, self-attention-based vision trans-formers (ViTs) have been adopted. Unlike CNNs, ViTs are heavy-weight. In this paper, we ask the following question: is it possible to combine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks? Towards this end, we introduce MobileViT, a light-weight and general-purpose vision transformer for mobile devices. MobileViT presents a different perspective for the global processing of information with transformers, i.e., transformers as convolutions. Our results show that MobileViT significantly outperforms CNN- and ViT-based networks across different tasks and datasets. On the ImageNet-1k dataset, MobileViT achieves top-1 accuracy of 78.4% with about 6 million parameters, which is 3.2% and 6.2% more accurate than MobileNetv3 (CNN-based) and DeIT (ViT-based) for a similar number of parameters. On the MS-COCO object detection task, MobileViT is 5.7% more accurate than MobileNetv3 for a similar number of parameters.*\n\nThis model was contributed by [matthijs](https://huggingface.co/Matthijs). The TensorFlow version of the model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code and weights can be found [here](https://github.com/apple/ml-cvnets).\n\n## Usage tips\n\n- MobileViT is more like a CNN than a Transformer model. It does not work on sequence data but on batches of images. Unlike ViT, there are no embeddings. The backbone model outputs a feature map. You can follow [this tutorial](https://keras.io/examples/vision/mobilevit) for a lightweight introduction.\n- One can use [`MobileViTImageProcessor`] to prepare images for the model. Note that if you do your own preprocessing, the pretrained checkpoints expect images to be in BGR pixel order (not RGB).\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes).\n- The segmentation model uses a [DeepLabV3](https://arxiv.org/abs/1706.05587) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).\n- As the name suggests MobileViT was designed to be performant and efficient on mobile phones. The TensorFlow versions of the MobileViT models are fully compatible with [TensorFlow Lite](https://www.tensorflow.org/lite).\n\n  You can use the following code to convert a MobileViT checkpoint (be it image classification or semantic segmentation) to generate a\n  TensorFlow Lite model:\n\n```py\nfrom transformers import TFMobileViTForImageClassification\nimport tensorflow as tf"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7b2638ff-cead-4eb5-b0b5-ca3eea20fdd8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilevit", "title": "MobileViT", "tokens": 1539, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7b2638ff-cead-4eb5-b0b5-ca3eea20fdd8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a5aed73d-b889-4ae8-902f-4943066d3c7e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2a6a7626dcd1e35b5340b766066bbe0dbfa17288134bebbef7f2179372837e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5a5b2dee-5552-4924-9191-4fed639146e5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"091ef4a334f6e3826ddb0bc0e0056a2e737c045e901951bd589952712f4334d9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3366, \"end_char_idx\": 5761, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a5aed73d-b889-4ae8-902f-4943066d3c7e", "doc_id": "a5aed73d-b889-4ae8-902f-4943066d3c7e", "ref_doc_id": "a5aed73d-b889-4ae8-902f-4943066d3c7e", "chroma:document": "model_ckpt = \"apple/mobilevit-xx-small\"\nmodel = TFMobileViTForImageClassification.from_pretrained(model_ckpt)\n\nconverter = tf.lite.TFLiteConverter.from_keras_model(model)\nconverter.optimizations = [tf.lite.Optimize.DEFAULT]\nconverter.target_spec.supported_ops = [\n    tf.lite.OpsSet.TFLITE_BUILTINS,\n    tf.lite.OpsSet.SELECT_TF_OPS,\n]\ntflite_model = converter.convert()\ntflite_filename = model_ckpt.split(\"/\")[-1] + \".tflite\"\nwith open(tflite_filename, \"wb\") as f:\n    f.write(tflite_model)\n```\n\n  The resulting model will be just **about an MB** making it a good fit for mobile applications where resources and network\n  bandwidth can be constrained.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with MobileViT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`MobileViTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n**Semantic segmentation**\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## MobileViTConfig\n\n[[autodoc]] MobileViTConfig\n\n## MobileViTFeatureExtractor\n\n[[autodoc]] MobileViTFeatureExtractor\n    - __call__\n    - post_process_semantic_segmentation\n\n## MobileViTImageProcessor\n\n[[autodoc]] MobileViTImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n<frameworkcontent>\n<pt>\n\n## MobileViTModel\n\n[[autodoc]] MobileViTModel\n    - forward\n\n## MobileViTForImageClassification\n\n[[autodoc]] MobileViTForImageClassification\n    - forward\n\n## MobileViTForSemanticSegmentation\n\n[[autodoc]] MobileViTForSemanticSegmentation\n    - forward\n\n</pt>\n<tf>\n\n## TFMobileViTModel\n\n[[autodoc]] TFMobileViTModel\n    - call\n\n## TFMobileViTForImageClassification\n\n[[autodoc]] TFMobileViTForImageClassification\n    - call\n\n## TFMobileViTForSemanticSegmentation\n\n[[autodoc]] TFMobileViTForSemanticSegmentation\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5a2adc5c-cc74-4297-a904-038204d3601f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm", "title": "XLM", "tokens": 1480, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5a2adc5c-cc74-4297-a904-038204d3601f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cee226f5-4caa-4cb4-b62d-e06022696fd6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f291245267f20043c181e90db571acc6ad405927b96309c6e5498a4c715d618b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5f92f2d5-180c-4cb3-ab0a-007330ad34d1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ec4aab5bc9c8b14dd3b87b2572c264e59b70ff05301238fd618e8f81b1b4f0d3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3207, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cee226f5-4caa-4cb4-b62d-e06022696fd6", "doc_id": "cee226f5-4caa-4cb4-b62d-e06022696fd6", "ref_doc_id": "cee226f5-4caa-4cb4-b62d-e06022696fd6", "chroma:document": "# XLM\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=xlm\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-xlm-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/xlm-mlm-en-2048\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe XLM model was proposed in [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by\nGuillaume Lample, Alexis Conneau. It's a transformer pretrained using one of the following objectives:\n\n- a causal language modeling (CLM) objective (next token prediction),\n- a masked language modeling (MLM) objective (BERT-like), or\n- a Translation Language Modeling (TLM) object (extension of BERT's MLM to multiple language inputs)\n\nThe abstract from the paper is the following:\n\n*Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding.\nIn this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We\npropose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual\ndata, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain\nstate-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our\napproach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we\nobtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised\nmachine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the\nprevious best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/facebookresearch/XLM/).\n\n## Usage tips\n\n- XLM has many different checkpoints, which were trained using different objectives: CLM, MLM or TLM. Make sure to\n  select the correct objective for your task (e.g. MLM checkpoints are not suitable for generation).\n- XLM has multilingual checkpoints which leverage a specific `lang` parameter. Check out the [multi-lingual](../multilingual) page for more information.\n- A transformer model trained on several languages. There are three different type of training for this model and the library provides checkpoints for all of them:\n\n    * Causal language modeling (CLM) which is the traditional autoregressive training (so this model could be in the previous section as well). One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages.\n    * Masked language modeling (MLM) which is like RoBERTa. One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages, with dynamic masking of the tokens."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5f92f2d5-180c-4cb3-ab0a-007330ad34d1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm", "title": "XLM", "tokens": 1480, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5f92f2d5-180c-4cb3-ab0a-007330ad34d1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cee226f5-4caa-4cb4-b62d-e06022696fd6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f291245267f20043c181e90db571acc6ad405927b96309c6e5498a4c715d618b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5a2adc5c-cc74-4297-a904-038204d3601f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"65c72d0c62e0574d4822cba9827154f9bbf7153df5cb0db5aa40c011e1f288b5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8a6eb603-6286-4996-97dd-40c75e2de0e8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7b5675865909b5c551c15eaf437fc64b5042ca674d8d399e543ef31e2f59799\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1537, \"end_char_idx\": 3502, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cee226f5-4caa-4cb4-b62d-e06022696fd6", "doc_id": "cee226f5-4caa-4cb4-b62d-e06022696fd6", "ref_doc_id": "cee226f5-4caa-4cb4-b62d-e06022696fd6", "chroma:document": "On unsupervised machine translation, we\nobtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised\nmachine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the\nprevious best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/facebookresearch/XLM/).\n\n## Usage tips\n\n- XLM has many different checkpoints, which were trained using different objectives: CLM, MLM or TLM. Make sure to\n  select the correct objective for your task (e.g. MLM checkpoints are not suitable for generation).\n- XLM has multilingual checkpoints which leverage a specific `lang` parameter. Check out the [multi-lingual](../multilingual) page for more information.\n- A transformer model trained on several languages. There are three different type of training for this model and the library provides checkpoints for all of them:\n\n    * Causal language modeling (CLM) which is the traditional autoregressive training (so this model could be in the previous section as well). One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages.\n    * Masked language modeling (MLM) which is like RoBERTa. One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages, with dynamic masking of the tokens.\n    * A combination of MLM and translation language modeling (TLM). This consists of concatenating a sentence in two different languages, with random masking. To predict one of the masked tokens, the model can use both, the surrounding context in language 1 and the context given by language 2."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8a6eb603-6286-4996-97dd-40c75e2de0e8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm", "title": "XLM", "tokens": 1480, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8a6eb603-6286-4996-97dd-40c75e2de0e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cee226f5-4caa-4cb4-b62d-e06022696fd6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f291245267f20043c181e90db571acc6ad405927b96309c6e5498a4c715d618b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5f92f2d5-180c-4cb3-ab0a-007330ad34d1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b2d05f8044b94ee5af1c607e520797bed61a85a93c84d06d322acaf1be6a320a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1852, \"end_char_idx\": 5279, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cee226f5-4caa-4cb4-b62d-e06022696fd6", "doc_id": "cee226f5-4caa-4cb4-b62d-e06022696fd6", "ref_doc_id": "cee226f5-4caa-4cb4-b62d-e06022696fd6", "chroma:document": "Our code and pretrained models will be made publicly available.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/facebookresearch/XLM/).\n\n## Usage tips\n\n- XLM has many different checkpoints, which were trained using different objectives: CLM, MLM or TLM. Make sure to\n  select the correct objective for your task (e.g. MLM checkpoints are not suitable for generation).\n- XLM has multilingual checkpoints which leverage a specific `lang` parameter. Check out the [multi-lingual](../multilingual) page for more information.\n- A transformer model trained on several languages. There are three different type of training for this model and the library provides checkpoints for all of them:\n\n    * Causal language modeling (CLM) which is the traditional autoregressive training (so this model could be in the previous section as well). One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages.\n    * Masked language modeling (MLM) which is like RoBERTa. One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages, with dynamic masking of the tokens.\n    * A combination of MLM and translation language modeling (TLM). This consists of concatenating a sentence in two different languages, with random masking. To predict one of the masked tokens, the model can use both, the surrounding context in language 1 and the context given by language 2.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## XLMConfig\n\n[[autodoc]] XLMConfig\n\n## XLMTokenizer\n\n[[autodoc]] XLMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## XLM specific outputs\n\n[[autodoc]] models.xlm.modeling_xlm.XLMForQuestionAnsweringOutput\n\n<frameworkcontent>\n<pt>\n\n## XLMModel\n\n[[autodoc]] XLMModel\n    - forward\n\n## XLMWithLMHeadModel\n\n[[autodoc]] XLMWithLMHeadModel\n    - forward\n\n## XLMForSequenceClassification\n\n[[autodoc]] XLMForSequenceClassification\n    - forward\n\n## XLMForMultipleChoice\n\n[[autodoc]] XLMForMultipleChoice\n    - forward\n\n## XLMForTokenClassification\n\n[[autodoc]] XLMForTokenClassification\n    - forward\n\n## XLMForQuestionAnsweringSimple\n\n[[autodoc]] XLMForQuestionAnsweringSimple\n    - forward\n\n## XLMForQuestionAnswering\n\n[[autodoc]] XLMForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFXLMModel\n\n[[autodoc]] TFXLMModel\n    - call\n\n## TFXLMWithLMHeadModel\n\n[[autodoc]] TFXLMWithLMHeadModel\n    - call\n\n## TFXLMForSequenceClassification\n\n[[autodoc]] TFXLMForSequenceClassification\n    - call\n\n## TFXLMForMultipleChoice\n\n[[autodoc]] TFXLMForMultipleChoice\n    - call\n\n## TFXLMForTokenClassification\n\n[[autodoc]] TFXLMForTokenClassification\n    - call\n\n## TFXLMForQuestionAnsweringSimple\n\n[[autodoc]] TFXLMForQuestionAnsweringSimple\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 45783d10-a110-46c3-822c-2bf34e46d475
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longt5", "title": "LongT5", "tokens": 1472, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"45783d10-a110-46c3-822c-2bf34e46d475\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c08dba81-682a-4976-a10f-f8035f6955f5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"68a76385445061cde8285ffd29f0ca81005c69b8896d504d0a32ed823abfc012\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ccfd1046-8667-4113-89bb-20af4b7a92f5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db10f259194fbc66791fbba6109c540b40916e2a1e7ddd1a60756f52d14a21a4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3414, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c08dba81-682a-4976-a10f-f8035f6955f5", "doc_id": "c08dba81-682a-4976-a10f-f8035f6955f5", "ref_doc_id": "c08dba81-682a-4976-a10f-f8035f6955f5", "chroma:document": "# LongT5\n\n## Overview\n\nThe LongT5 model was proposed in [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://arxiv.org/abs/2112.07916)\nby Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung and Yinfei Yang. It's an\nencoder-decoder transformer pre-trained in a text-to-text denoising generative setting. LongT5 model is an extension of\nT5 model, and it enables using one of the two different efficient attention mechanisms - (1) Local attention, or (2)\nTransient-Global attention.\n\n\nThe abstract from the paper is the following:\n\n*Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the\nperformance of Transformer-based neural models. In this paper, we present a new model, called LongT5, with which we\nexplore the effects of scaling both the input length and model size at the same time. Specifically, we integrated\nattention ideas from long-input transformers (ETC), and adopted pre-training strategies from summarization pre-training\n(PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call {\\em Transient Global}\n(TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are\nable to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on\nquestion answering tasks.*\n\nThis model was contributed by [stancld](https://huggingface.co/stancld).\nThe original code can be found [here](https://github.com/google-research/longt5).\n\n## Usage tips\n\n- [`LongT5ForConditionalGeneration`] is an extension of [`T5ForConditionalGeneration`] exchanging the traditional\nencoder *self-attention* layer with efficient either *local* attention or *transient-global* (*tglobal*) attention.\n- Unlike the T5 model, LongT5 does not use a task prefix. Furthermore, it uses a different pre-training objective\ninspired by the pre-training of [`PegasusForConditionalGeneration`].\n- LongT5 model is designed to work efficiently and very well on long-range *sequence-to-sequence* tasks where the\ninput sequence exceeds commonly used 512 tokens. It is capable of handling input sequences of a length up to 16,384 tokens.\n- For *Local Attention*, the sparse sliding-window local attention operation allows a given token to attend only `r`\ntokens to the left and right of it (with `r=127` by default). *Local Attention* does not introduce any new parameters\nto the model. The complexity of the mechanism is linear in input sequence length `l`: `O(l*r)`.\n- *Transient Global Attention* is an extension of the *Local Attention*. It, furthermore, allows each input token to\ninteract with all other tokens in the layer. This is achieved via splitting an input sequence into blocks of a fixed\nlength `k` (with a default `k=16`). Then, a global token for such a block is obtained via summing and normalizing the embeddings of every token\nin the block. Thanks to this, the attention allows each token to attend to both nearby tokens like in Local attention, and\nalso every global token like in the case of standard global attention (*transient* represents the fact the global tokens\nare constructed dynamically within each attention operation).  As a consequence, *TGlobal* attention introduces\na few new parameters -- global relative position biases and a layer normalization for global token's embedding."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ccfd1046-8667-4113-89bb-20af4b7a92f5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longt5", "title": "LongT5", "tokens": 1472, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ccfd1046-8667-4113-89bb-20af4b7a92f5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c08dba81-682a-4976-a10f-f8035f6955f5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"68a76385445061cde8285ffd29f0ca81005c69b8896d504d0a32ed823abfc012\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45783d10-a110-46c3-822c-2bf34e46d475\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80e65468c7f6f401d2b59fece27c546673541daa50d88752c205e6c5ef6fb26e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"41de031c-15a0-4b13-a124-9d659b447610\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1e22f244a32719f6bc6e71a4017b705709bf670d95814f32e91e277861b9f49f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1817, \"end_char_idx\": 4833, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c08dba81-682a-4976-a10f-f8035f6955f5", "doc_id": "c08dba81-682a-4976-a10f-f8035f6955f5", "ref_doc_id": "c08dba81-682a-4976-a10f-f8035f6955f5", "chroma:document": "- Unlike the T5 model, LongT5 does not use a task prefix. Furthermore, it uses a different pre-training objective\ninspired by the pre-training of [`PegasusForConditionalGeneration`].\n- LongT5 model is designed to work efficiently and very well on long-range *sequence-to-sequence* tasks where the\ninput sequence exceeds commonly used 512 tokens. It is capable of handling input sequences of a length up to 16,384 tokens.\n- For *Local Attention*, the sparse sliding-window local attention operation allows a given token to attend only `r`\ntokens to the left and right of it (with `r=127` by default). *Local Attention* does not introduce any new parameters\nto the model. The complexity of the mechanism is linear in input sequence length `l`: `O(l*r)`.\n- *Transient Global Attention* is an extension of the *Local Attention*. It, furthermore, allows each input token to\ninteract with all other tokens in the layer. This is achieved via splitting an input sequence into blocks of a fixed\nlength `k` (with a default `k=16`). Then, a global token for such a block is obtained via summing and normalizing the embeddings of every token\nin the block. Thanks to this, the attention allows each token to attend to both nearby tokens like in Local attention, and\nalso every global token like in the case of standard global attention (*transient* represents the fact the global tokens\nare constructed dynamically within each attention operation).  As a consequence, *TGlobal* attention introduces\na few new parameters -- global relative position biases and a layer normalization for global token's embedding.\nThe complexity of this mechanism is `O(l(r + l/k))`.\n- An example showing how to evaluate a fine-tuned LongT5 model on the [pubmed dataset](https://huggingface.co/datasets/scientific_papers) is below.\n\n```python\n>>> import evaluate\n>>> from datasets import load_dataset\n>>> from transformers import AutoTokenizer, LongT5ForConditionalGeneration\n\n>>> dataset = load_dataset(\"scientific_papers\", \"pubmed\", split=\"validation\")\n>>> model = (\n...     LongT5ForConditionalGeneration.from_pretrained(\"Stancld/longt5-tglobal-large-16384-pubmed-3k_steps\")\n...     .to(\"cuda\")\n...     .half()\n... )\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Stancld/longt5-tglobal-large-16384-pubmed-3k_steps\")\n\n\n>>> def generate_answers(batch):\n...     inputs_dict = tokenizer(\n...         batch[\"article\"], max_length=16384, padding=\"max_length\", truncation=True, return_tensors=\"pt\"\n...     )\n...     input_ids = inputs_dict.input_ids.to(\"cuda\")\n...     attention_mask = inputs_dict.attention_mask.to(\"cuda\")\n...     output_ids = model.generate(input_ids, attention_mask=attention_mask, max_length=512, num_beams=2)\n...     batch[\"predicted_abstract\"] = tokenizer.batch_decode(output_ids, skip_special_tokens=True)\n...     return batch\n\n\n>>> result = dataset.map(generate_answer, batched=True, batch_size=2)\n>>> rouge = evaluate.load(\"rouge\")\n>>> rouge.compute(predictions=result[\"predicted_abstract\"], references=result[\"abstract\"])\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 41de031c-15a0-4b13-a124-9d659b447610
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longt5", "title": "LongT5", "tokens": 1472, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"41de031c-15a0-4b13-a124-9d659b447610\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c08dba81-682a-4976-a10f-f8035f6955f5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"68a76385445061cde8285ffd29f0ca81005c69b8896d504d0a32ed823abfc012\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ccfd1046-8667-4113-89bb-20af4b7a92f5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2d19f14e891c2e1d430221ac0794cf17c50a0a2a5c1b137376f36fd4b1b1b71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3254, \"end_char_idx\": 5495, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c08dba81-682a-4976-a10f-f8035f6955f5", "doc_id": "c08dba81-682a-4976-a10f-f8035f6955f5", "ref_doc_id": "c08dba81-682a-4976-a10f-f8035f6955f5", "chroma:document": "As a consequence, *TGlobal* attention introduces\na few new parameters -- global relative position biases and a layer normalization for global token's embedding.\nThe complexity of this mechanism is `O(l(r + l/k))`.\n- An example showing how to evaluate a fine-tuned LongT5 model on the [pubmed dataset](https://huggingface.co/datasets/scientific_papers) is below.\n\n```python\n>>> import evaluate\n>>> from datasets import load_dataset\n>>> from transformers import AutoTokenizer, LongT5ForConditionalGeneration\n\n>>> dataset = load_dataset(\"scientific_papers\", \"pubmed\", split=\"validation\")\n>>> model = (\n...     LongT5ForConditionalGeneration.from_pretrained(\"Stancld/longt5-tglobal-large-16384-pubmed-3k_steps\")\n...     .to(\"cuda\")\n...     .half()\n... )\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Stancld/longt5-tglobal-large-16384-pubmed-3k_steps\")\n\n\n>>> def generate_answers(batch):\n...     inputs_dict = tokenizer(\n...         batch[\"article\"], max_length=16384, padding=\"max_length\", truncation=True, return_tensors=\"pt\"\n...     )\n...     input_ids = inputs_dict.input_ids.to(\"cuda\")\n...     attention_mask = inputs_dict.attention_mask.to(\"cuda\")\n...     output_ids = model.generate(input_ids, attention_mask=attention_mask, max_length=512, num_beams=2)\n...     batch[\"predicted_abstract\"] = tokenizer.batch_decode(output_ids, skip_special_tokens=True)\n...     return batch\n\n\n>>> result = dataset.map(generate_answer, batched=True, batch_size=2)\n>>> rouge = evaluate.load(\"rouge\")\n>>> rouge.compute(predictions=result[\"predicted_abstract\"], references=result[\"abstract\"])\n```\n\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## LongT5Config\n\n[[autodoc]] LongT5Config\n\n<frameworkcontent>\n<pt>\n\n## LongT5Model\n\n[[autodoc]] LongT5Model\n    - forward\n\n## LongT5ForConditionalGeneration\n\n[[autodoc]] LongT5ForConditionalGeneration\n    - forward\n\n## LongT5EncoderModel\n\n[[autodoc]] LongT5EncoderModel\n    - forward\n\n</pt>\n<jax>\n\n## FlaxLongT5Model\n\n[[autodoc]] FlaxLongT5Model\n    - __call__\n    - encode\n    - decode\n\n## FlaxLongT5ForConditionalGeneration\n\n[[autodoc]] FlaxLongT5ForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ddf96090-5fc3-49f0-b8b0-c54f1400ea86
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cpm", "title": "CPM", "tokens": 670, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ddf96090-5fc3-49f0-b8b0-c54f1400ea86\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cpm\", \"title\": \"CPM\", \"tokens\": 670, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e879b53f-07b3-4b40-82da-2c86f387d75d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cpm\", \"title\": \"CPM\", \"tokens\": 670, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5cd54ffcdd4c1079dbc39fab2cdf38116ea25acd1aaf113469e4a5f846566aa0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1931, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e879b53f-07b3-4b40-82da-2c86f387d75d", "doc_id": "e879b53f-07b3-4b40-82da-2c86f387d75d", "ref_doc_id": "e879b53f-07b3-4b40-82da-2c86f387d75d", "chroma:document": "# CPM\n\n## Overview\n\nThe CPM model was proposed in [CPM: A Large-scale Generative Chinese Pre-trained Language Model](https://arxiv.org/abs/2012.00413) by Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin,\nYusheng Su, Haozhe Ji, Jian Guan, Fanchao Qi, Xiaozhi Wang, Yanan Zheng, Guoyang Zeng, Huanqi Cao, Shengqi Chen,\nDaixuan Li, Zhenbo Sun, Zhiyuan Liu, Minlie Huang, Wentao Han, Jie Tang, Juanzi Li, Xiaoyan Zhu, Maosong Sun.\n\nThe abstract from the paper is the following:\n\n*Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3,\nwith 175 billion parameters and 570GB training data, drew a lot of attention due to the capacity of few-shot (even\nzero-shot) learning. However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus\nof GPT-3 is primarily English, and the parameters are not publicly available. In this technical report, we release the\nChinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. To the best\nof our knowledge, CPM, with 2.6 billion parameters and 100GB Chinese training data, is the largest Chinese pre-trained\nlanguage model, which could facilitate several downstream Chinese NLP tasks, such as conversation, essay generation,\ncloze test, and language understanding. Extensive experiments demonstrate that CPM achieves strong performance on many\nNLP tasks in the settings of few-shot (even zero-shot) learning.*\n\nThis model was contributed by [canwenxu](https://huggingface.co/canwenxu). The original implementation can be found\nhere: https://github.com/TsinghuaAI/CPM-Generate\n\n\n<Tip>\n\nCPM's architecture is the same as GPT-2, except for tokenization method. Refer to [GPT-2 documentation](gpt2) for \nAPI reference information.  \n\n</Tip>\n\n\n## CpmTokenizer\n\n[[autodoc]] CpmTokenizer\n\n## CpmTokenizerFast\n\n[[autodoc]] CpmTokenizerFast"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8333cb40-92a1-4347-b24d-aeac38c2c042
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/patchtst", "title": "PatchTST", "tokens": 871, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8333cb40-92a1-4347-b24d-aeac38c2c042\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtst\", \"title\": \"PatchTST\", \"tokens\": 871, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3590c793-045e-48de-981f-2b14f1247f48\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtst\", \"title\": \"PatchTST\", \"tokens\": 871, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"334974bbfb741d76f27267cb6100117fa0e3c91a81a5ac7dae6971136b4fa8b9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2975, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3590c793-045e-48de-981f-2b14f1247f48", "doc_id": "3590c793-045e-48de-981f-2b14f1247f48", "ref_doc_id": "3590c793-045e-48de-981f-2b14f1247f48", "chroma:document": "# PatchTST\n\n## Overview\n\nThe PatchTST model was proposed in [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://arxiv.org/abs/2211.14730) by Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong and Jayant Kalagnanam.\n\nAt a high level the model vectorizes time series into patches of a given size and encodes the resulting sequence of vectors via a Transformer that then outputs the prediction length forecast via an appropriate head. The model is illustrated in the following figure:\n\n![model](https://github.com/namctin/transformers/assets/8100/150af169-29de-419a-8d98-eb78251c21fa)\n\nThe abstract from the paper is the following:\n\n*We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy.*\n\nThis model was contributed by [namctin](https://huggingface.co/namctin), [gsinthong](https://huggingface.co/gsinthong), [diepi](https://huggingface.co/diepi), [vijaye12](https://huggingface.co/vijaye12), [wmgifford](https://huggingface.co/wmgifford), and [kashif](https://huggingface.co/kashif). The original code can be found [here](https://github.com/yuqinie98/PatchTST).\n\n## Usage tips\n\nThe model can also be used for time series classification and time series regression. See the respective [`PatchTSTForClassification`] and [`PatchTSTForRegression`] classes.\n\n## Resources\n\n- A blog post explaining PatchTST in depth can be found [here](https://huggingface.co/blog/patchtst). The blog can also be opened in Google Colab.\n\n## PatchTSTConfig\n\n[[autodoc]] PatchTSTConfig\n\n## PatchTSTModel\n\n[[autodoc]] PatchTSTModel\n    - forward\n\n## PatchTSTForPrediction\n\n[[autodoc]] PatchTSTForPrediction\n    - forward\n\n## PatchTSTForClassification\n\n[[autodoc]] PatchTSTForClassification\n    - forward\n\n## PatchTSTForPretraining\n\n[[autodoc]] PatchTSTForPretraining\n    - forward\n\n## PatchTSTForRegression\n\n[[autodoc]] PatchTSTForRegression\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7aa62d81-11db-4055-9912-f88126890504
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longformer", "title": "Longformer", "tokens": 2001, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7aa62d81-11db-4055-9912-f88126890504\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4bbda101-80e7-42f1-8cb8-6ab8ff0481fe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1b101f4a3e9c49d0c6a2a27510d3fa148fc242e347775d53d294e12790b22ef3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cd5294c8-4730-43c3-9f5e-32fb68e99806\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"510eec8c6680600eff0cbe37b6e0ef70b2ed66efbc3c6875cbbeb58bac370967\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3103, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "doc_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "ref_doc_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "chroma:document": "# Longformer\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=longformer\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-longformer-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/longformer-base-4096-finetuned-squadv1\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Longformer model was presented in [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf) by Iz Beltagy, Matthew E. Peters, Arman Cohan.\n\nThe abstract from the paper is the following:\n\n*Transformer-based models are unable to process long sequences due to their self-attention operation, which scales\nquadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention\nmechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or\nlonger. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local\nwindowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we\nevaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In\ncontrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our\npretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on\nWikiHop and TriviaQA.*\n\nThis model was contributed by [beltagy](https://huggingface.co/beltagy). The Authors' code can be found [here](https://github.com/allenai/longformer).\n\n## Usage tips\n\n- Since the Longformer is based on RoBERTa, it doesn't have `token_type_ids`. You don't need to indicate which\n  token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or\n  `</s>`).\n- A transformer model replacing the attention matrices by sparse matrices to go faster. Often, the local context (e.g., what are the two tokens left and right?) is enough to take action for a given token. Some preselected input tokens are still given global attention, but the attention matrix has way less parameters, resulting in a speed-up. See the local attention section for more information.\n\n## Longformer Self Attention\n\nLongformer self attention employs self attention on both a \"local\" context and a \"global\" context. Most tokens only\nattend \"locally\" to each other meaning that each token attends to its \\\\(\\frac{1}{2} w\\\\) previous tokens and\n\\\\(\\frac{1}{2} w\\\\) succeeding tokens with \\\\(w\\\\) being the window length as defined in\n`config.attention_window`. Note that `config.attention_window` can be of type `List` to define a\ndifferent \\\\(w\\\\) for each layer. A selected few tokens attend \"globally\" to all other tokens, as it is\nconventionally done for all tokens in `BertSelfAttention`.\n\nNote that \"locally\" and \"globally\" attending tokens are projected by different query, key and value matrices."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cd5294c8-4730-43c3-9f5e-32fb68e99806
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longformer", "title": "Longformer", "tokens": 2001, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cd5294c8-4730-43c3-9f5e-32fb68e99806\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4bbda101-80e7-42f1-8cb8-6ab8ff0481fe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1b101f4a3e9c49d0c6a2a27510d3fa148fc242e347775d53d294e12790b22ef3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7aa62d81-11db-4055-9912-f88126890504\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c26ea8b3baed7fa9f8fdb532716ba94d8f8da9f828091dfdaae6ae1cad182b9f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5deed0b0-ba83-40f5-8825-4c8bdb606661\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69c2e8679c53872313ba6f3643743da5e0eb9d58428d6ccf8bb0542fcdbd6cbd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "doc_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "ref_doc_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "chroma:document": "*\n\nThis model was contributed by [beltagy](https://huggingface.co/beltagy). The Authors' code can be found [here](https://github.com/allenai/longformer).\n\n## Usage tips\n\n- Since the Longformer is based on RoBERTa, it doesn't have `token_type_ids`. You don't need to indicate which\n  token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or\n  `</s>`).\n- A transformer model replacing the attention matrices by sparse matrices to go faster. Often, the local context (e.g., what are the two tokens left and right?) is enough to take action for a given token. Some preselected input tokens are still given global attention, but the attention matrix has way less parameters, resulting in a speed-up. See the local attention section for more information.\n\n## Longformer Self Attention\n\nLongformer self attention employs self attention on both a \"local\" context and a \"global\" context. Most tokens only\nattend \"locally\" to each other meaning that each token attends to its \\\\(\\frac{1}{2} w\\\\) previous tokens and\n\\\\(\\frac{1}{2} w\\\\) succeeding tokens with \\\\(w\\\\) being the window length as defined in\n`config.attention_window`. Note that `config.attention_window` can be of type `List` to define a\ndifferent \\\\(w\\\\) for each layer. A selected few tokens attend \"globally\" to all other tokens, as it is\nconventionally done for all tokens in `BertSelfAttention`.\n\nNote that \"locally\" and \"globally\" attending tokens are projected by different query, key and value matrices. Also note\nthat every \"locally\" attending token not only attends to tokens within its window \\\\(w\\\\), but also to all \"globally\"\nattending tokens so that global attention is *symmetric*.\n\nThe user can define which tokens attend \"locally\" and which tokens attend \"globally\" by setting the tensor\n`global_attention_mask` at run-time appropriately. All Longformer models employ the following logic for\n`global_attention_mask`:\n\n- 0: the token attends \"locally\",\n- 1: the token attends \"globally\".\n\nFor more information please also refer to [`~LongformerModel.forward`] method.\n\nUsing Longformer self attention, the memory and time complexity of the query-key matmul operation, which usually\nrepresents the memory and time bottleneck, can be reduced from \\\\(\\mathcal{O}(n_s \\times n_s)\\\\) to\n\\\\(\\mathcal{O}(n_s \\times w)\\\\), with \\\\(n_s\\\\) being the sequence length and \\\\(w\\\\) being the average window\nsize. It is assumed that the number of \"globally\" attending tokens is insignificant as compared to the number of\n\"locally\" attending tokens.\n\nFor more information, please refer to the official [paper](https://arxiv.org/pdf/2004.05150.pdf).\n\n\n## Training\n\n[`LongformerForMaskedLM`] is trained the exact same way [`RobertaForMaskedLM`] is\ntrained and should be used as follows:\n\n```python\ninput_ids = tokenizer.encode(\"This is a sentence from [MASK] training data\", return_tensors=\"pt\")\nmlm_labels = tokenizer.encode(\"This is a sentence from the training data\", return_tensors=\"pt\")\n\nloss = model(input_ids, labels=input_ids, masked_lm_labels=mlm_labels)[0]\n```\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5deed0b0-ba83-40f5-8825-4c8bdb606661
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longformer", "title": "Longformer", "tokens": 2001, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5deed0b0-ba83-40f5-8825-4c8bdb606661\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4bbda101-80e7-42f1-8cb8-6ab8ff0481fe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1b101f4a3e9c49d0c6a2a27510d3fa148fc242e347775d53d294e12790b22ef3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cd5294c8-4730-43c3-9f5e-32fb68e99806\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"54fdcd19b3c789bcd315408efff916ee61793ac2220b497614cebc04f479adcb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"982d483f-e718-4e6e-881a-7df427258d9b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6b9995a05e5b5dbcdd4e82c2e083ecdb8b64923315e0ed5ba31793969c690886\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "doc_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "ref_doc_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "chroma:document": "The user can define which tokens attend \"locally\" and which tokens attend \"globally\" by setting the tensor\n`global_attention_mask` at run-time appropriately. All Longformer models employ the following logic for\n`global_attention_mask`:\n\n- 0: the token attends \"locally\",\n- 1: the token attends \"globally\".\n\nFor more information please also refer to [`~LongformerModel.forward`] method.\n\nUsing Longformer self attention, the memory and time complexity of the query-key matmul operation, which usually\nrepresents the memory and time bottleneck, can be reduced from \\\\(\\mathcal{O}(n_s \\times n_s)\\\\) to\n\\\\(\\mathcal{O}(n_s \\times w)\\\\), with \\\\(n_s\\\\) being the sequence length and \\\\(w\\\\) being the average window\nsize. It is assumed that the number of \"globally\" attending tokens is insignificant as compared to the number of\n\"locally\" attending tokens.\n\nFor more information, please refer to the official [paper](https://arxiv.org/pdf/2004.05150.pdf).\n\n\n## Training\n\n[`LongformerForMaskedLM`] is trained the exact same way [`RobertaForMaskedLM`] is\ntrained and should be used as follows:\n\n```python\ninput_ids = tokenizer.encode(\"This is a sentence from [MASK] training data\", return_tensors=\"pt\")\nmlm_labels = tokenizer.encode(\"This is a sentence from the training data\", return_tensors=\"pt\")\n\nloss = model(input_ids, labels=input_ids, masked_lm_labels=mlm_labels)[0]\n```\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## LongformerConfig\n\n[[autodoc]] LongformerConfig\n\n## LongformerTokenizer\n\n[[autodoc]] LongformerTokenizer\n\n## LongformerTokenizerFast\n\n[[autodoc]] LongformerTokenizerFast\n\n## Longformer specific outputs\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerBaseModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerBaseModelOutputWithPooling\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerMaskedLMOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerSequenceClassifierOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerTokenClassifierOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerBaseModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerBaseModelOutputWithPooling\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerMaskedLMOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerQuestionAnsweringModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerSequenceClassifierOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerMultipleChoiceModelOutput\n\n[[autodoc]] models.longformer."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 982d483f-e718-4e6e-881a-7df427258d9b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longformer", "title": "Longformer", "tokens": 2001, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"982d483f-e718-4e6e-881a-7df427258d9b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4bbda101-80e7-42f1-8cb8-6ab8ff0481fe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1b101f4a3e9c49d0c6a2a27510d3fa148fc242e347775d53d294e12790b22ef3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5deed0b0-ba83-40f5-8825-4c8bdb606661\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"060ac3ba1cebc61b0bffdc803d584b06b38385050ac1c298584c375c1971f9ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "doc_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "ref_doc_id": "4bbda101-80e7-42f1-8cb8-6ab8ff0481fe", "chroma:document": "/tasks/question_answering)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## LongformerConfig\n\n[[autodoc]] LongformerConfig\n\n## LongformerTokenizer\n\n[[autodoc]] LongformerTokenizer\n\n## LongformerTokenizerFast\n\n[[autodoc]] LongformerTokenizerFast\n\n## Longformer specific outputs\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerBaseModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerBaseModelOutputWithPooling\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerMaskedLMOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerSequenceClassifierOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerTokenClassifierOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerBaseModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerBaseModelOutputWithPooling\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerMaskedLMOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerQuestionAnsweringModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerSequenceClassifierOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerMultipleChoiceModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerTokenClassifierOutput\n\n<frameworkcontent>\n<pt>\n\n## LongformerModel\n\n[[autodoc]] LongformerModel\n    - forward\n\n## LongformerForMaskedLM\n\n[[autodoc]] LongformerForMaskedLM\n    - forward\n\n## LongformerForSequenceClassification\n\n[[autodoc]] LongformerForSequenceClassification\n    - forward\n\n## LongformerForMultipleChoice\n\n[[autodoc]] LongformerForMultipleChoice\n    - forward\n\n## LongformerForTokenClassification\n\n[[autodoc]] LongformerForTokenClassification\n    - forward\n\n## LongformerForQuestionAnswering\n\n[[autodoc]] LongformerForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFLongformerModel\n\n[[autodoc]] TFLongformerModel\n    - call\n\n## TFLongformerForMaskedLM\n\n[[autodoc]] TFLongformerForMaskedLM\n    - call\n\n## TFLongformerForQuestionAnswering\n\n[[autodoc]] TFLongformerForQuestionAnswering\n    - call\n\n## TFLongformerForSequenceClassification\n\n[[autodoc]] TFLongformerForSequenceClassification\n    - call\n\n## TFLongformerForTokenClassification\n\n[[autodoc]] TFLongformerForTokenClassification\n    - call\n\n## TFLongformerForMultipleChoice\n\n[[autodoc]] TFLongformerForMultipleChoice\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 298acbc3-afaa-461e-b4d7-0af98233482a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/groupvit", "title": "GroupViT", "tokens": 1049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"298acbc3-afaa-461e-b4d7-0af98233482a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dcbc5eb9-7d77-4600-9295-3d6a47f6c9c3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1872af444c2ff9259be2ed06fa160c8d744ef0df2b30043911cf43c5bd40dfca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3efab360-f86e-432c-8840-8cc464042c3b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"febc131d565aa02af5d24c558f1e82b6fbc94ead8c583767174fa405c40855fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2742, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dcbc5eb9-7d77-4600-9295-3d6a47f6c9c3", "doc_id": "dcbc5eb9-7d77-4600-9295-3d6a47f6c9c3", "ref_doc_id": "dcbc5eb9-7d77-4600-9295-3d6a47f6c9c3", "chroma:document": "# GroupViT\n\n## Overview\n\nThe GroupViT model was proposed in [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://arxiv.org/abs/2202.11094) by Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang.\nInspired by [CLIP](clip), GroupViT is a vision-language model that can perform zero-shot semantic segmentation on any given vocabulary categories.\n\nThe abstract from the paper is the following:\n\n*Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision.*\n\nThis model was contributed by [xvjiarui](https://huggingface.co/xvjiarui). The TensorFlow version was contributed by [ariG23498](https://huggingface.co/ariG23498) with the help of [Yih-Dar SHIEH](https://huggingface.co/ydshieh), [Amy Roberts](https://huggingface.co/amyeroberts), and [Joao Gante](https://huggingface.co/joaogante).\nThe original code can be found [here](https://github.com/NVlabs/GroupViT).\n\n## Usage tips\n \n- You may specify `output_segmentation=True` in the forward of `GroupViTModel` to get the segmentation logits of input texts. \n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GroupViT.\n\n- The quickest way to get started with GroupViT is by checking the [example notebooks](https://github.com/xvjiarui/GroupViT/blob/main/demo/GroupViT_hf_inference_notebook.ipynb) (which showcase zero-shot segmentation inference).\n- One can also check out the [HuggingFace Spaces demo](https://huggingface.co/spaces/xvjiarui/GroupViT) to play with GroupViT."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3efab360-f86e-432c-8840-8cc464042c3b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/groupvit", "title": "GroupViT", "tokens": 1049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3efab360-f86e-432c-8840-8cc464042c3b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dcbc5eb9-7d77-4600-9295-3d6a47f6c9c3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1872af444c2ff9259be2ed06fa160c8d744ef0df2b30043911cf43c5bd40dfca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"298acbc3-afaa-461e-b4d7-0af98233482a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"250d81a13e324bb85e404bb486b8872b6e21e7ceccd7bc34d47c990c578cca24\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1235, \"end_char_idx\": 3494, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dcbc5eb9-7d77-4600-9295-3d6a47f6c9c3", "doc_id": "dcbc5eb9-7d77-4600-9295-3d6a47f6c9c3", "ref_doc_id": "dcbc5eb9-7d77-4600-9295-3d6a47f6c9c3", "chroma:document": "With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision.*\n\nThis model was contributed by [xvjiarui](https://huggingface.co/xvjiarui). The TensorFlow version was contributed by [ariG23498](https://huggingface.co/ariG23498) with the help of [Yih-Dar SHIEH](https://huggingface.co/ydshieh), [Amy Roberts](https://huggingface.co/amyeroberts), and [Joao Gante](https://huggingface.co/joaogante).\nThe original code can be found [here](https://github.com/NVlabs/GroupViT).\n\n## Usage tips\n \n- You may specify `output_segmentation=True` in the forward of `GroupViTModel` to get the segmentation logits of input texts. \n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GroupViT.\n\n- The quickest way to get started with GroupViT is by checking the [example notebooks](https://github.com/xvjiarui/GroupViT/blob/main/demo/GroupViT_hf_inference_notebook.ipynb) (which showcase zero-shot segmentation inference).\n- One can also check out the [HuggingFace Spaces demo](https://huggingface.co/spaces/xvjiarui/GroupViT) to play with GroupViT. \n\n## GroupViTConfig\n\n[[autodoc]] GroupViTConfig\n    - from_text_vision_configs\n\n## GroupViTTextConfig\n\n[[autodoc]] GroupViTTextConfig\n\n## GroupViTVisionConfig\n\n[[autodoc]] GroupViTVisionConfig\n\n<frameworkcontent>\n<pt>\n\n## GroupViTModel\n\n[[autodoc]] GroupViTModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## GroupViTTextModel\n\n[[autodoc]] GroupViTTextModel\n    - forward\n\n## GroupViTVisionModel\n\n[[autodoc]] GroupViTVisionModel\n    - forward\n\n</pt>\n<tf>\n\n## TFGroupViTModel\n\n[[autodoc]] TFGroupViTModel\n    - call\n    - get_text_features\n    - get_image_features\n\n## TFGroupViTTextModel\n\n[[autodoc]] TFGroupViTTextModel\n    - call\n\n## TFGroupViTVisionModel\n\n[[autodoc]] TFGroupViTVisionModel\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4c45232a-3c4d-4746-9c06-e1ab3559b794
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pix2struct", "title": "Pix2Struct", "tokens": 943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4c45232a-3c4d-4746-9c06-e1ab3559b794\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pix2struct\", \"title\": \"Pix2Struct\", \"tokens\": 943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e06d7c4c-94c2-49b4-b6ca-cdea49776938\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pix2struct\", \"title\": \"Pix2Struct\", \"tokens\": 943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f83a13f234516d0009395133681354697c3014e87f6bc7f3c1e5b6a290811b1c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3455, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e06d7c4c-94c2-49b4-b6ca-cdea49776938", "doc_id": "e06d7c4c-94c2-49b4-b6ca-cdea49776938", "ref_doc_id": "e06d7c4c-94c2-49b4-b6ca-cdea49776938", "chroma:document": "# Pix2Struct\n\n## Overview\n\nThe Pix2Struct model was proposed in [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://arxiv.org/abs/2210.03347) by Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova.\n\nThe abstract from the paper is the following:\n\n> Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to this diversity, previous work has typically relied on domain-specific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy, we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions are rendered directly on top of the input image. For the first time, we show that a single pretrained model can achieve state-of-the-art results in six out of nine tasks across four domains: documents, illustrations, user interfaces, and natural images.\n\nTips:\n\nPix2Struct has been fine tuned on a variety of tasks and datasets, ranging from image captioning, visual question answering (VQA) over different inputs (books, charts, science diagrams), captioning UI components etc. The full list can be found in Table 1 of the paper.\nWe therefore advise you to use these models for the tasks they have been fine tuned on. For instance, if you want to use Pix2Struct for UI captioning, you should use the model fine tuned on the UI dataset. If you want to use Pix2Struct for image captioning, you should use the model fine tuned on the natural images captioning dataset and so on.\n\nIf you want to use the model to perform conditional text captioning, make sure to use the processor with `add_special_tokens=False`.\n\nThis model was contributed by [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/google-research/pix2struct).\n\n## Resources\n\n- [Fine-tuning Notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb)\n- [All models](https://huggingface.co/models?search=pix2struct)\n\n## Pix2StructConfig\n\n[[autodoc]] Pix2StructConfig\n    - from_text_vision_configs\n\n## Pix2StructTextConfig\n\n[[autodoc]] Pix2StructTextConfig\n\n## Pix2StructVisionConfig\n\n[[autodoc]] Pix2StructVisionConfig\n\n## Pix2StructProcessor\n\n[[autodoc]] Pix2StructProcessor\n\n## Pix2StructImageProcessor\n\n[[autodoc]] Pix2StructImageProcessor\n    - preprocess\n\n## Pix2StructTextModel\n\n[[autodoc]] Pix2StructTextModel\n    - forward\n\n## Pix2StructVisionModel\n\n[[autodoc]] Pix2StructVisionModel\n    - forward\n\n## Pix2StructForConditionalGeneration\n\n[[autodoc]] Pix2StructForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7bccfa4e-a664-4bf9-a558-5a09b0c18651
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mamba2", "title": "Mamba 2", "tokens": 1437, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7bccfa4e-a664-4bf9-a558-5a09b0c18651\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba2\", \"title\": \"Mamba 2\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7b115d75-e989-4a35-96ee-beca8f6dd320\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba2\", \"title\": \"Mamba 2\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2a06a996375f6dd6abed12e0a0117c78a1eea96f559c52c1e9a3f089e65c55cf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f011c75b-a186-43b3-afdb-c6f60b53d356\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9493e4c6a84808a1bc81de94bc68ac0db100c9e678989b50d6b5f423ccaf0485\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3280, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7b115d75-e989-4a35-96ee-beca8f6dd320", "doc_id": "7b115d75-e989-4a35-96ee-beca8f6dd320", "ref_doc_id": "7b115d75-e989-4a35-96ee-beca8f6dd320", "chroma:document": "# Mamba 2\n\n## Overview\n\nThe Mamba2 model was proposed in [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060) by Tri Dao and Albert Gu. It is a State Space Model similar to Mamba 1, with better performances in a simplified architecture. \n\n\nThe abstract from the paper is the following:\n\n*While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices. Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.*\n\nTips:\n\nThis version should support all implementations of Mamba 2, and in particular [Mamba-2 codestral](https://huggingface.co/mistralai/Mamba-Codestral-7B-v0.1) from Mistral AI. In particular, mamba 2 codestral was released with a number of `groups` equal to 8, which can be thought intuitively as similar to the number of kv heads in an attention-based model. \nThis model has two different forward passes, `torch_forward` or `cuda_kernels_forward`. The latter uses the original cuda kernels if they are found in your environment, and is slower on the prefill i.e. requires a \"warmup run\" due to high cpu overhead, see [here](https://github.com/state-spaces/mamba/issues/389#issuecomment-2171755306) and [also here](https://github.com/state-spaces/mamba/issues/355#issuecomment-2147597457). Without compilation, the `torch_forward` implementation is faster by a factor 3 to 4. Further, there are no positional embeddings in this model, but there is an `attention_mask` and a specific logic to mask out hidden states in two places in the case of batched generation, see [here](https://github.com/state-spaces/mamba/issues/66#issuecomment-1863563829) as well. Due to this, in addition to the reimplementation of mamba2 kernels, batched generation and cached generation are expected to have slight discrepancies. Further, the results given by the cuda kernels or the torch forward are expected to be slightly different. The SSM algorithm heavily relies on tensor contractions, which have matmul equivalents but the order of operations is slightly different, making the difference greater at smaller precisions. \nAnother note, shutdown of hidden states corresponding to padding tokens is done in 2 places and mostly has been tested with left-padding. Right-padding will propagate noise down the line and is not guaranteed to yield satisfactory results. `tokenizer.padding_side = \"left\"` ensures you are using the correct padding side.\n\nThis model was contributed by [Molbap](https://huggingface.co/Molbap), with tremendous help from [Anton Vlasjuk](https://github.com/vasqu).\nThe original code can be found [here](https://github.com/state-spaces/mamba)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f011c75b-a186-43b3-afdb-c6f60b53d356
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mamba2", "title": "Mamba 2", "tokens": 1437, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f011c75b-a186-43b3-afdb-c6f60b53d356\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba2\", \"title\": \"Mamba 2\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7b115d75-e989-4a35-96ee-beca8f6dd320\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba2\", \"title\": \"Mamba 2\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2a06a996375f6dd6abed12e0a0117c78a1eea96f559c52c1e9a3f089e65c55cf\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7bccfa4e-a664-4bf9-a558-5a09b0c18651\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mamba2\", \"title\": \"Mamba 2\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ed2cca8e4f24af72d96aa33c5a6cb804d5415dcf3742f981d57a5c75be8fc512\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3283, \"end_char_idx\": 5239, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7b115d75-e989-4a35-96ee-beca8f6dd320", "doc_id": "7b115d75-e989-4a35-96ee-beca8f6dd320", "ref_doc_id": "7b115d75-e989-4a35-96ee-beca8f6dd320", "chroma:document": "# Usage\n\n### A simple generation example: \n```python \nfrom transformers import Mamba2Config, Mamba2ForCausalLM, AutoTokenizer\nimport torch\nmodel_id = 'mistralai/Mamba-Codestral-7B-v0.1'\ntokenizer = AutoTokenizer.from_pretrained(model_id, revision='refs/pr/9', from_slow=True, legacy=False)\nmodel = Mamba2ForCausalLM.from_pretrained(model_id, revision='refs/pr/9')\ninput_ids = tokenizer(\"Hey how are you doing?\", return_tensors= \"pt\")[\"input_ids\"]\n\nout = model.generate(input_ids, max_new_tokens=10)\nprint(tokenizer.batch_decode(out))\n```\n\nHere's a draft script for finetuning: \n```python \nfrom trl import SFTTrainer\nfrom peft import LoraConfig\nfrom transformers import AutoTokenizer, Mamba2ForCausalLM, TrainingArguments\nmodel_id = 'mistralai/Mamba-Codestral-7B-v0.1'\ntokenizer = AutoTokenizer.from_pretrained(model_id, revision='refs/pr/9', from_slow=True, legacy=False)\ntokenizer.pad_token = tokenizer.eos_token\ntokenizer.padding_side = \"left\" #enforce padding side left\n\nmodel = Mamba2ForCausalLM.from_pretrained(model_id, revision='refs/pr/9')\ndataset = load_dataset(\"Abirate/english_quotes\", split=\"train\")\n# Without CUDA kernels, batch size of 2 occupies one 80GB device\n# but precision can be reduced.\n# Experiments and trials welcome!\ntraining_args = TrainingArguments(\n    output_dir=\"./results\",\n    num_train_epochs=3,\n    per_device_train_batch_size=2,\n    logging_dir='./logs',\n    logging_steps=10,\n    learning_rate=2e-3\n)\nlora_config =  LoraConfig(\n        r=8,\n        target_modules=[\"embeddings\", \"in_proj\", \"out_proj\"],\n        task_type=\"CAUSAL_LM\",\n        bias=\"none\"\n)\ntrainer = SFTTrainer(\n    model=model,\n    tokenizer=tokenizer,\n    args=training_args,\n    peft_config=lora_config,\n    train_dataset=dataset,\n    dataset_text_field=\"quote\",\n)\ntrainer.train()\n```\n\n\n## Mamba2Config\n\n[[autodoc]] Mamba2Config\n\n## Mamba2Model\n\n[[autodoc]] Mamba2Model\n    - forward\n\n## Mamba2LMHeadModel\n\n[[autodoc]] Mamba2ForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 12da704e-8c99-40d5-8940-41a3633e0462
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"12da704e-8c99-40d5-8940-41a3633e0462\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bd032317-7053-4db0-a599-81823b8c6938\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"63dd5889-f7a7-4c62-a439-c23418cd7292\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3fe4070a9d580384c2feca594af49ba9c9af58fb2af77503f1af5dd922fb893c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2096, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bd032317-7053-4db0-a599-81823b8c6938", "doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "ref_doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "chroma:document": "# AWQ\n\n<Tip>\n\nTry AWQ quantization with this [notebook](https://colab.research.google.com/drive/1HzZH89yAXJaZgwJDhQj9LqSBux932BvY)!\n\n</Tip>\n\n[Activation-aware Weight Quantization (AWQ)](https://hf.co/papers/2306.00978) doesn't quantize all the weights in a model, and instead, it preserves a small percentage of weights that are important for LLM performance. This significantly reduces quantization loss such that you can run models in 4-bit precision without experiencing any performance degradation.\n\nThere are several libraries for quantizing models with the AWQ algorithm, such as [llm-awq](https://github.com/mit-han-lab/llm-awq), [autoawq](https://github.com/casper-hansen/AutoAWQ) or [optimum-intel](https://huggingface.co/docs/optimum/main/en/intel/optimization_inc). Transformers supports loading models quantized with the llm-awq and autoawq libraries. This guide will show you how to load models quantized with autoawq, but the process is similar for llm-awq quantized models.\n\nMake sure you have autoawq installed:\n\n```bash\npip install autoawq\n```\n\nAWQ-quantized models can be identified by checking the `quantization_config` attribute in the model's [config.json](https://huggingface.co/TheBloke/zephyr-7B-alpha-AWQ/blob/main/config.json) file:\n\n```json\n{\n  \"_name_or_path\": \"/workspace/process/huggingfaceh4_zephyr-7b-alpha/source\",\n  \"architectures\": [\n    \"MistralForCausalLM\"\n  ],\n  ...\n  ...\n  ...\n  \"quantization_config\": {\n    \"quant_method\": \"awq\",\n    \"zero_point\": true,\n    \"group_size\": 128,\n    \"bits\": 4,\n    \"version\": \"gemm\"\n  }\n}\n```\n\nA quantized model is loaded with the [`~PreTrainedModel.from_pretrained`] method. If you loaded your model on the CPU, make sure to move it to a GPU device first. Use the `device_map` parameter to specify where to place the model:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"TheBloke/zephyr-7B-alpha-AWQ\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"cuda:0\")\n```\n\nLoading an AWQ-quantized model automatically sets other weights to fp16 by default for performance reasons."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 63dd5889-f7a7-4c62-a439-c23418cd7292
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"63dd5889-f7a7-4c62-a439-c23418cd7292\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bd032317-7053-4db0-a599-81823b8c6938\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"12da704e-8c99-40d5-8940-41a3633e0462\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bb97af195acdca8b43b95e62005ffc3aa85847daaab6154afcfd17ab06f2dece\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"306adf3c-2c47-47cf-b238-933bf6dad69a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1558afb1780de96a9ae0fa16c205db5b4514bbe62e1b45e7a233b84f347bf4ca\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 777, \"end_char_idx\": 3709, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bd032317-7053-4db0-a599-81823b8c6938", "doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "ref_doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "chroma:document": "Transformers supports loading models quantized with the llm-awq and autoawq libraries. This guide will show you how to load models quantized with autoawq, but the process is similar for llm-awq quantized models.\n\nMake sure you have autoawq installed:\n\n```bash\npip install autoawq\n```\n\nAWQ-quantized models can be identified by checking the `quantization_config` attribute in the model's [config.json](https://huggingface.co/TheBloke/zephyr-7B-alpha-AWQ/blob/main/config.json) file:\n\n```json\n{\n  \"_name_or_path\": \"/workspace/process/huggingfaceh4_zephyr-7b-alpha/source\",\n  \"architectures\": [\n    \"MistralForCausalLM\"\n  ],\n  ...\n  ...\n  ...\n  \"quantization_config\": {\n    \"quant_method\": \"awq\",\n    \"zero_point\": true,\n    \"group_size\": 128,\n    \"bits\": 4,\n    \"version\": \"gemm\"\n  }\n}\n```\n\nA quantized model is loaded with the [`~PreTrainedModel.from_pretrained`] method. If you loaded your model on the CPU, make sure to move it to a GPU device first. Use the `device_map` parameter to specify where to place the model:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"TheBloke/zephyr-7B-alpha-AWQ\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"cuda:0\")\n```\n\nLoading an AWQ-quantized model automatically sets other weights to fp16 by default for performance reasons. If you want to load these other weights in a different format, use the `torch_dtype` parameter:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"TheBloke/zephyr-7B-alpha-AWQ\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32)\n```\n\nAWQ quantization can also be combined with [FlashAttention-2](../perf_infer_gpu_one#flashattention-2) to further accelerate inference:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"TheBloke/zephyr-7B-alpha-AWQ\", attn_implementation=\"flash_attention_2\", device_map=\"cuda:0\")\n```\n\n## Fused modules\n\nFused modules offers improved accuracy and performance and it is supported out-of-the-box for AWQ modules for [Llama](https://huggingface.co/meta-llama) and [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) architectures, but you can also fuse AWQ modules for unsupported architectures.\n\n<Tip warning={true}>\n\nFused modules cannot be combined with other optimization techniques such as FlashAttention-2.\n\n</Tip>\n\n<hfoptions id=\"fuse\">\n<hfoption id=\"supported architectures\">\n\nTo enable fused modules for supported architectures, create an [`AwqConfig`] and set the parameters `fuse_max_seq_len` and `do_fuse=True`. The `fuse_max_seq_len` parameter is the total sequence length and it should include the context length and the expected generation length. You can set it to a larger value to be safe.\n\nFor example, to fuse the AWQ modules of the [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 306adf3c-2c47-47cf-b238-933bf6dad69a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"306adf3c-2c47-47cf-b238-933bf6dad69a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bd032317-7053-4db0-a599-81823b8c6938\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"63dd5889-f7a7-4c62-a439-c23418cd7292\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b542e8b1930d455227b4368704d4794caf5a296e206ddab4af69ada966098fca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b528476e-efda-47a1-a14c-107da9047f76\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e35a2e7ef5dead90955e49fb8f28462a3b7d2bd4eb0839774b28e980fc4e47aa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3055, \"end_char_idx\": 4207, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bd032317-7053-4db0-a599-81823b8c6938", "doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "ref_doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "chroma:document": "<Tip warning={true}>\n\nFused modules cannot be combined with other optimization techniques such as FlashAttention-2.\n\n</Tip>\n\n<hfoptions id=\"fuse\">\n<hfoption id=\"supported architectures\">\n\nTo enable fused modules for supported architectures, create an [`AwqConfig`] and set the parameters `fuse_max_seq_len` and `do_fuse=True`. The `fuse_max_seq_len` parameter is the total sequence length and it should include the context length and the expected generation length. You can set it to a larger value to be safe.\n\nFor example, to fuse the AWQ modules of the [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model.\n\n```python\nimport torch\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nmodel_id = \"TheBloke/Mistral-7B-OpenOrca-AWQ\"\n\nquantization_config = AwqConfig(\n    bits=4,\n    fuse_max_seq_len=512,\n    do_fuse=True,\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config).to(0)\n```\n\nThe [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model was benchmarked with `batch_size=1` with and without fused modules."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b528476e-efda-47a1-a14c-107da9047f76
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b528476e-efda-47a1-a14c-107da9047f76\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bd032317-7053-4db0-a599-81823b8c6938\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"306adf3c-2c47-47cf-b238-933bf6dad69a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"982c1d9a6c77db8550c28dc2333f1cf4cb7ebf3e0865c4ca4a0c558fc8041b0a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2909c5e8-873a-4d7f-8df8-4ce8ee2214a7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"458962dd4681edb7a1845c0d968b2b9584953dfec1dc96cf6b1ca96313124c10\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3055, \"end_char_idx\": 6567, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bd032317-7053-4db0-a599-81823b8c6938", "doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "ref_doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "chroma:document": "<Tip warning={true}>\n\nFused modules cannot be combined with other optimization techniques such as FlashAttention-2.\n\n</Tip>\n\n<hfoptions id=\"fuse\">\n<hfoption id=\"supported architectures\">\n\nTo enable fused modules for supported architectures, create an [`AwqConfig`] and set the parameters `fuse_max_seq_len` and `do_fuse=True`. The `fuse_max_seq_len` parameter is the total sequence length and it should include the context length and the expected generation length. You can set it to a larger value to be safe.\n\nFor example, to fuse the AWQ modules of the [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model.\n\n```python\nimport torch\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nmodel_id = \"TheBloke/Mistral-7B-OpenOrca-AWQ\"\n\nquantization_config = AwqConfig(\n    bits=4,\n    fuse_max_seq_len=512,\n    do_fuse=True,\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config).to(0)\n```\n\nThe [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model was benchmarked with `batch_size=1` with and without fused modules.\n\n<figcaption class=\"text-center text-gray-500 text-lg\">Unfused module</figcaption>\n\n|   Batch Size |   Prefill Length |   Decode Length |   Prefill tokens/s |   Decode tokens/s | Memory (VRAM)   |\n|-------------:|-----------------:|----------------:|-------------------:|------------------:|:----------------|\n|            1 |               32 |              32 |            60.0984 |           38.4537 | 4.50 GB (5.68%) |\n|            1 |               64 |              64 |          1333.67   |           31.6604 | 4.50 GB (5.68%) |\n|            1 |              128 |             128 |          2434.06   |           31.6272 | 4.50 GB (5.68%) |\n|            1 |              256 |             256 |          3072.26   |           38.1731 | 4.50 GB (5.68%) |\n|            1 |              512 |             512 |          3184.74   |           31.6819 | 4.59 GB (5.80%) |\n|            1 |             1024 |            1024 |          3148.18   |           36.8031 | 4.81 GB (6.07%) |\n|            1 |             2048 |            2048 |          2927.33   |           35.2676 | 5.73 GB (7.23%) |\n\n<figcaption class=\"text-center text-gray-500 text-lg\">Fused module</figcaption>\n\n|   Batch Size |   Prefill Length |   Decode Length |   Prefill tokens/s |   Decode tokens/s | Memory (VRAM)   |\n|-------------:|-----------------:|----------------:|-------------------:|------------------:|:----------------|\n|            1 |               32 |              32 |            81.4899 |           80.2569 | 4.00 GB (5.05%) |\n|            1 |               64 |              64 |          1756.1    |          106.26   | 4.00 GB (5.05%) |\n|            1 |              128 |             128 |          2479.32   |          105.631  | 4.00 GB (5.06%) |\n|            1 |              256 |             256 |          1813.6    |           85.7485 | 4.01 GB (5.06%) |\n|            1 |              512 |             512 |          2848.9    |           97.701  | 4.11 GB (5.19%) |\n|            1 |             1024 |            1024 |          3044.35   |           87.7323 | 4.41 GB (5.57%) |\n|            1 |             2048 |            2048 |          2715.11   |           89.4709 | 5.57 GB (7.04%) |\n\nThe speed and throughput of fused and unfused modules were also tested with the [optimum-benchmark](https://github.com/huggingface/optimum-benchmark) library."}


