# embeddings_queue

created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 90c1e0e7-637f-4897-9945-c722c1a509a0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/openai-gpt", "title": "OpenAI GPT", "tokens": 2009, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"90c1e0e7-637f-4897-9945-c722c1a509a0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"385c3cd9-a1dd-414b-85c4-3d5f04fb059e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af0e726e3f0aa2f1d1d817bb8d5c0114dadb17d9dbd055bf9d3736932ea8ea91\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c2534eb5-50d2-456a-a948-7bfa0a2bc266\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3936c892a571caaefb80a73e4d33634a3d4e657637dae0248e9fc759ed063438\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9a6c5ede-3cae-4b37-9a33-c5ea8c8adcb5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5831133e5fd41688d37ffec6e6b9836542dfba4dd92f061344c7072138d94283\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4180, \"end_char_idx\": 6148, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "385c3cd9-a1dd-414b-85c4-3d5f04fb059e", "doc_id": "385c3cd9-a1dd-414b-85c4-3d5f04fb059e", "ref_doc_id": "385c3cd9-a1dd-414b-85c4-3d5f04fb059e", "chroma:document": "- A blog on [Training CodeParrot \ud83e\udd9c from Scratch](https://huggingface.co/blog/codeparrot), a large GPT-2 model.\n- A blog on [Faster Text Generation with TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate) with GPT-2.\n- A blog on [How to train a Language Model with Megatron-LM](https://huggingface.co/blog/megatron-training) with a GPT-2 model.\n- A notebook on how to [finetune GPT2 to generate lyrics in the style of your favorite artist](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). \ud83c\udf0e\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`OpenAIGPTLMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-generation/run_generation.py) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFOpenAIGPTLMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- See also: [Causal language modeling task guide](../tasks/language_modeling)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A course material on [Byte-Pair Encoding tokenization](https://huggingface.co/course/en/chapter6/5)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9a6c5ede-3cae-4b37-9a33-c5ea8c8adcb5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/openai-gpt", "title": "OpenAI GPT", "tokens": 2009, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9a6c5ede-3cae-4b37-9a33-c5ea8c8adcb5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"385c3cd9-a1dd-414b-85c4-3d5f04fb059e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af0e726e3f0aa2f1d1d817bb8d5c0114dadb17d9dbd055bf9d3736932ea8ea91\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"90c1e0e7-637f-4897-9945-c722c1a509a0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/openai-gpt\", \"title\": \"OpenAI GPT\", \"tokens\": 2009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"09ed59f80e21e44dbb573a5fe5b3978e1cb09f7b07045663de57270509191365\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4743, \"end_char_idx\": 7224, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "385c3cd9-a1dd-414b-85c4-3d5f04fb059e", "doc_id": "385c3cd9-a1dd-414b-85c4-3d5f04fb059e", "ref_doc_id": "385c3cd9-a1dd-414b-85c4-3d5f04fb059e", "chroma:document": "\ud83c\udf0e\n- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). \ud83c\udf0e\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`OpenAIGPTLMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-generation/run_generation.py) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFOpenAIGPTLMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- See also: [Causal language modeling task guide](../tasks/language_modeling)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A course material on [Byte-Pair Encoding tokenization](https://huggingface.co/course/en/chapter6/5).\n\n## OpenAIGPTConfig\n\n[[autodoc]] OpenAIGPTConfig\n\n## OpenAIGPTTokenizer\n\n[[autodoc]] OpenAIGPTTokenizer\n    - save_vocabulary\n\n## OpenAIGPTTokenizerFast\n\n[[autodoc]] OpenAIGPTTokenizerFast\n\n## OpenAI specific outputs\n\n[[autodoc]] models.openai.modeling_openai.OpenAIGPTDoubleHeadsModelOutput\n\n[[autodoc]] models.openai.modeling_tf_openai.TFOpenAIGPTDoubleHeadsModelOutput\n\n<frameworkcontent>\n<pt>\n\n## OpenAIGPTModel\n\n[[autodoc]] OpenAIGPTModel\n    - forward\n\n## OpenAIGPTLMHeadModel\n\n[[autodoc]] OpenAIGPTLMHeadModel\n    - forward\n\n## OpenAIGPTDoubleHeadsModel\n\n[[autodoc]] OpenAIGPTDoubleHeadsModel\n    - forward\n\n## OpenAIGPTForSequenceClassification\n\n[[autodoc]] OpenAIGPTForSequenceClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFOpenAIGPTModel\n\n[[autodoc]] TFOpenAIGPTModel\n    - call\n\n## TFOpenAIGPTLMHeadModel\n\n[[autodoc]] TFOpenAIGPTLMHeadModel\n    - call\n\n## TFOpenAIGPTDoubleHeadsModel\n\n[[autodoc]] TFOpenAIGPTDoubleHeadsModel\n    - call\n\n## TFOpenAIGPTForSequenceClassification\n\n[[autodoc]] TFOpenAIGPTForSequenceClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7ed513d1-535f-43bb-8c42-3c76c0c3fc36
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/levit", "title": "LeViT", "tokens": 1578, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7ed513d1-535f-43bb-8c42-3c76c0c3fc36\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"46be66cf-1ced-46dd-adec-444d6e4f5edc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08ddfd157b56f771c9af038fc5f787b347fffa193bcb5d305d60af400ea94fff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c75357c0-4636-4285-92e9-ff826330ccd7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ed2cce1163764d34d065d8fc141fbfca552170835c3a66fb0f4ef5947a430e93\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "46be66cf-1ced-46dd-adec-444d6e4f5edc", "doc_id": "46be66cf-1ced-46dd-adec-444d6e4f5edc", "ref_doc_id": "46be66cf-1ced-46dd-adec-444d6e4f5edc", "chroma:document": "# LeViT\n\n## Overview\n\nThe LeViT model was proposed in [LeViT: Introducing Convolutions to Vision Transformers](https://arxiv.org/abs/2104.01136) by Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv\u00e9 J\u00e9gou, Matthijs Douze. LeViT improves the [Vision Transformer (ViT)](vit) in performance and efficiency by a few architectural differences such as activation maps with decreasing resolutions in Transformers and the introduction of an attention bias to integrate positional information.\n\nThe abstract from the paper is the following:\n\n*We design a family of image classification architectures that optimize the trade-off between accuracy\nand efficiency in a high-speed regime. Our work exploits recent findings in attention-based architectures,\nwhich are competitive on highly parallel processing hardware. We revisit principles from the extensive\nliterature on convolutional neural networks to apply them to transformers, in particular activation maps\nwith decreasing resolutions. We also introduce the attention bias, a new way to integrate positional information\nin vision transformers. As a result, we propose LeVIT: a hybrid neural network for fast inference image classification.\nWe consider different measures of efficiency on different hardware platforms, so as to best reflect a wide range of\napplication scenarios. Our extensive experiments empirically validate our technical choices and show they are suitable\nto most architectures. Overall, LeViT significantly outperforms existing convnets and vision transformers with respect\nto the speed/accuracy tradeoff. For example, at 80% ImageNet top-1 accuracy, LeViT is 5 times faster than EfficientNet on CPU. *\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/levit_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> LeViT Architecture. Taken from the <a href=\"https://arxiv.org/abs/2104.01136\">original paper</a>.</small>\n\nThis model was contributed by [anugunj](https://huggingface.co/anugunj). The original code can be found [here](https://github.com/facebookresearch/LeViT).\n\n## Usage tips\n\n- Compared to ViT, LeViT models use an additional distillation head to effectively learn from a teacher (which, in the LeViT paper, is a ResNet like-model). The distillation head is learned through backpropagation under supervision of a ResNet like-model. They also draw inspiration from convolution neural networks to use activation maps with decreasing resolutions to increase the efficiency.\n- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top\n  of the final hidden state and not using the distillation head, or (2) by placing both a prediction head and distillation\n  head on top of the final hidden state. In that case, the prediction head is trained using regular cross-entropy between\n  the prediction of the head and the ground-truth label, while the distillation prediction head is trained using hard distillation\n  (cross-entropy between the prediction of the distillation head and the label predicted by the teacher). At inference time,\n  one takes the average prediction between both heads as final prediction. (2) is also called \"fine-tuning with distillation\",\n  because one relies on a teacher that has already been fine-tuned on the downstream dataset. In terms of models, (1) corresponds\n  to [`LevitForImageClassification`] and (2) corresponds to [`LevitForImageClassificationWithTeacher`]."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c75357c0-4636-4285-92e9-ff826330ccd7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/levit", "title": "LeViT", "tokens": 1578, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c75357c0-4636-4285-92e9-ff826330ccd7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"46be66cf-1ced-46dd-adec-444d6e4f5edc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08ddfd157b56f771c9af038fc5f787b347fffa193bcb5d305d60af400ea94fff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7ed513d1-535f-43bb-8c42-3c76c0c3fc36\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a27a930a3a062f7af22c0f8f7a63949c0c4524782925977153ef143e7e90ea81\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"20ee3527-cf21-49d5-a1c4-6fca0c434b53\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ae41758e61637e18eaaf463a55e41a5fc36f38178757d9252f4c1a0e516b331\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1867, \"end_char_idx\": 4977, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "46be66cf-1ced-46dd-adec-444d6e4f5edc", "doc_id": "46be66cf-1ced-46dd-adec-444d6e4f5edc", "ref_doc_id": "46be66cf-1ced-46dd-adec-444d6e4f5edc", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2104.01136\">original paper</a>.</small>\n\nThis model was contributed by [anugunj](https://huggingface.co/anugunj). The original code can be found [here](https://github.com/facebookresearch/LeViT).\n\n## Usage tips\n\n- Compared to ViT, LeViT models use an additional distillation head to effectively learn from a teacher (which, in the LeViT paper, is a ResNet like-model). The distillation head is learned through backpropagation under supervision of a ResNet like-model. They also draw inspiration from convolution neural networks to use activation maps with decreasing resolutions to increase the efficiency.\n- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top\n  of the final hidden state and not using the distillation head, or (2) by placing both a prediction head and distillation\n  head on top of the final hidden state. In that case, the prediction head is trained using regular cross-entropy between\n  the prediction of the head and the ground-truth label, while the distillation prediction head is trained using hard distillation\n  (cross-entropy between the prediction of the distillation head and the label predicted by the teacher). At inference time,\n  one takes the average prediction between both heads as final prediction. (2) is also called \"fine-tuning with distillation\",\n  because one relies on a teacher that has already been fine-tuned on the downstream dataset. In terms of models, (1) corresponds\n  to [`LevitForImageClassification`] and (2) corresponds to [`LevitForImageClassificationWithTeacher`].\n- All released checkpoints were pre-trained and fine-tuned on  [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k)\n  (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). only. No external data was used. This is in\n  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for\n  pre-training.\n- The authors of LeViT released 5 trained LeViT models, which you can directly plug into [`LevitModel`] or [`LevitForImageClassification`].\n  Techniques like data augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset\n  (while only using ImageNet-1k for pre-training). The 5 variants available are (all trained on images of size 224x224):\n  *facebook/levit-128S*, *facebook/levit-128*, *facebook/levit-192*, *facebook/levit-256* and\n  *facebook/levit-384*. Note that one should use [`LevitImageProcessor`] in order to\n  prepare images for the model.\n- [`LevitForImageClassificationWithTeacher`] currently supports only inference and not training or fine-tuning.\n- You can check out demo notebooks regarding inference as well as fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer)\n  (you can just replace [`ViTFeatureExtractor`] by [`LevitImageProcessor`] and [`ViTForImageClassification`] by [`LevitForImageClassification`] or [`LevitForImageClassificationWithTeacher`])."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 20ee3527-cf21-49d5-a1c4-6fca0c434b53
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/levit", "title": "LeViT", "tokens": 1578, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"20ee3527-cf21-49d5-a1c4-6fca0c434b53\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"46be66cf-1ced-46dd-adec-444d6e4f5edc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08ddfd157b56f771c9af038fc5f787b347fffa193bcb5d305d60af400ea94fff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c75357c0-4636-4285-92e9-ff826330ccd7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/levit\", \"title\": \"LeViT\", \"tokens\": 1578, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3452ccef3947123c5777c05963c867affa70a7467bc335484171e2de8989693a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3502, \"end_char_idx\": 6188, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "46be66cf-1ced-46dd-adec-444d6e4f5edc", "doc_id": "46be66cf-1ced-46dd-adec-444d6e4f5edc", "ref_doc_id": "46be66cf-1ced-46dd-adec-444d6e4f5edc", "chroma:document": "- All released checkpoints were pre-trained and fine-tuned on  [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k)\n  (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). only. No external data was used. This is in\n  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for\n  pre-training.\n- The authors of LeViT released 5 trained LeViT models, which you can directly plug into [`LevitModel`] or [`LevitForImageClassification`].\n  Techniques like data augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset\n  (while only using ImageNet-1k for pre-training). The 5 variants available are (all trained on images of size 224x224):\n  *facebook/levit-128S*, *facebook/levit-128*, *facebook/levit-192*, *facebook/levit-256* and\n  *facebook/levit-384*. Note that one should use [`LevitImageProcessor`] in order to\n  prepare images for the model.\n- [`LevitForImageClassificationWithTeacher`] currently supports only inference and not training or fine-tuning.\n- You can check out demo notebooks regarding inference as well as fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer)\n  (you can just replace [`ViTFeatureExtractor`] by [`LevitImageProcessor`] and [`ViTForImageClassification`] by [`LevitForImageClassification`] or [`LevitForImageClassificationWithTeacher`]).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LeViT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`LevitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## LevitConfig\n\n[[autodoc]] LevitConfig\n\n## LevitFeatureExtractor\n\n[[autodoc]] LevitFeatureExtractor\n    - __call__\n\n## LevitImageProcessor\n\n  [[autodoc]] LevitImageProcessor\n    - preprocess\n\n## LevitModel\n\n[[autodoc]] LevitModel\n    - forward\n\n## LevitForImageClassification\n\n[[autodoc]] LevitForImageClassification\n    - forward\n\n## LevitForImageClassificationWithTeacher\n\n[[autodoc]] LevitForImageClassificationWithTeacher\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 53329dd3-9d49-41bf-aa6f-f5575a44bc1b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v2", "title": "MobileNet V2", "tokens": 1526, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"53329dd3-9d49-41bf-aa6f-f5575a44bc1b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"36c7c889-2b29-4bce-95f5-5ca92d6483c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"120e8ad1559fa83bc26316e7c5d0caaaf6fb8b41fcee7af3745aa72828a80c74\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dfc28f55-d4de-4171-af8b-14af3f14e712\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b86f4f7eaf5a1186616086a2ea84c2a97f70ce75d296891b4cc95fe30193579\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3361, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "36c7c889-2b29-4bce-95f5-5ca92d6483c6", "doc_id": "36c7c889-2b29-4bce-95f5-5ca92d6483c6", "ref_doc_id": "36c7c889-2b29-4bce-95f5-5ca92d6483c6", "chroma:document": "# MobileNet V2\n\n## Overview\n\nThe MobileNet model was proposed in [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://arxiv.org/abs/1801.04381) by Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen.\n\nThe abstract from the paper is the following:\n\n*In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3.*\n\n*The MobileNetV2 architecture is based on an inverted residual structure where the input and output of the residual block are thin bottleneck layers opposite to traditional residual models which use expanded representations in the input an MobileNetV2 uses lightweight depthwise convolutions to filter features in the intermediate expansion layer. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on Imagenet classification, COCO object detection, VOC image segmentation. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as the number of parameters.*\n\nThis model was contributed by [matthijs](https://huggingface.co/Matthijs). The original code and weights can be found [here for the main model](https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet) and [here for DeepLabV3+](https://github.com/tensorflow/models/tree/master/research/deeplab).\n\n## Usage tips\n\n- The checkpoints are named **mobilenet\\_v2\\_*depth*\\_*size***, for example **mobilenet\\_v2\\_1.0\\_224**, where **1.0** is the depth multiplier (sometimes also referred to as \"alpha\" or the width multiplier) and **224** is the resolution of the input images the model was trained on.\n\n- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.\n\n- One can use [`MobileNetV2ImageProcessor`] to prepare images for the model.\n\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra \u201cbackground\u201d class (index 0).\n\n- The segmentation model uses a [DeepLabV3+](https://arxiv.org/abs/1802.02611) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).\n\n- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: dfc28f55-d4de-4171-af8b-14af3f14e712
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v2", "title": "MobileNet V2", "tokens": 1526, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dfc28f55-d4de-4171-af8b-14af3f14e712\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"36c7c889-2b29-4bce-95f5-5ca92d6483c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"120e8ad1559fa83bc26316e7c5d0caaaf6fb8b41fcee7af3745aa72828a80c74\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"53329dd3-9d49-41bf-aa6f-f5575a44bc1b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ab9d694aa61a4fdea5278caaff1430c7d6f763dfa4e97b69093cc97c02246414\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ad9ebfc-a19a-4882-b68f-cc180758949d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9638d3cc2e017925b8999b907b772b33b30a82e487d7b83f92dced7945289c87\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1849, \"end_char_idx\": 5060, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "36c7c889-2b29-4bce-95f5-5ca92d6483c6", "doc_id": "36c7c889-2b29-4bce-95f5-5ca92d6483c6", "ref_doc_id": "36c7c889-2b29-4bce-95f5-5ca92d6483c6", "chroma:document": "The original code and weights can be found [here for the main model](https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet) and [here for DeepLabV3+](https://github.com/tensorflow/models/tree/master/research/deeplab).\n\n## Usage tips\n\n- The checkpoints are named **mobilenet\\_v2\\_*depth*\\_*size***, for example **mobilenet\\_v2\\_1.0\\_224**, where **1.0** is the depth multiplier (sometimes also referred to as \"alpha\" or the width multiplier) and **224** is the resolution of the input images the model was trained on.\n\n- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.\n\n- One can use [`MobileNetV2ImageProcessor`] to prepare images for the model.\n\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra \u201cbackground\u201d class (index 0).\n\n- The segmentation model uses a [DeepLabV3+](https://arxiv.org/abs/1802.02611) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).\n\n- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size. To use native PyTorch padding behavior, create a [`MobileNetV2Config`] with `tf_padding = False`.\n\nUnsupported features:\n\n- The [`MobileNetV2Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use an average pooling layer with a fixed 7x7 window and stride 1 instead of global pooling. For inputs that are larger than the recommended image size, this gives a pooled output that is larger than 1x1. The Hugging Face implementation does not support this.\n\n- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional \"FakeQuantization\" operations to unquantize the weights.\n\n- It's common to extract the output from the expansion layers at indices 10 and 13, as well as the output from the final 1x1 convolution layer, for downstream purposes. Using `output_hidden_states=True` returns the output from all intermediate layers. There is currently no way to limit this to specific layers.\n\n- The DeepLabV3+ segmentation head does not use the final convolution layer from the backbone, but this layer gets computed anyway. There is currently no way to tell [`MobileNetV2Model`] up to which layer it should run.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with MobileNetV2.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`MobileNetV2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1ad9ebfc-a19a-4882-b68f-cc180758949d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v2", "title": "MobileNet V2", "tokens": 1526, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1ad9ebfc-a19a-4882-b68f-cc180758949d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"36c7c889-2b29-4bce-95f5-5ca92d6483c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"120e8ad1559fa83bc26316e7c5d0caaaf6fb8b41fcee7af3745aa72828a80c74\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dfc28f55-d4de-4171-af8b-14af3f14e712\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v2\", \"title\": \"MobileNet V2\", \"tokens\": 1526, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e63799d2b8ae919d6b10295b27a2b49f42cb15a54f3f62df378905f6a6a5196c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3362, \"end_char_idx\": 6037, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "36c7c889-2b29-4bce-95f5-5ca92d6483c6", "doc_id": "36c7c889-2b29-4bce-95f5-5ca92d6483c6", "ref_doc_id": "36c7c889-2b29-4bce-95f5-5ca92d6483c6", "chroma:document": "To use native PyTorch padding behavior, create a [`MobileNetV2Config`] with `tf_padding = False`.\n\nUnsupported features:\n\n- The [`MobileNetV2Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use an average pooling layer with a fixed 7x7 window and stride 1 instead of global pooling. For inputs that are larger than the recommended image size, this gives a pooled output that is larger than 1x1. The Hugging Face implementation does not support this.\n\n- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional \"FakeQuantization\" operations to unquantize the weights.\n\n- It's common to extract the output from the expansion layers at indices 10 and 13, as well as the output from the final 1x1 convolution layer, for downstream purposes. Using `output_hidden_states=True` returns the output from all intermediate layers. There is currently no way to limit this to specific layers.\n\n- The DeepLabV3+ segmentation head does not use the final convolution layer from the backbone, but this layer gets computed anyway. There is currently no way to tell [`MobileNetV2Model`] up to which layer it should run.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with MobileNetV2.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`MobileNetV2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n**Semantic segmentation**\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## MobileNetV2Config\n\n[[autodoc]] MobileNetV2Config\n\n## MobileNetV2FeatureExtractor\n\n[[autodoc]] MobileNetV2FeatureExtractor\n    - preprocess\n    - post_process_semantic_segmentation\n\n## MobileNetV2ImageProcessor\n\n[[autodoc]] MobileNetV2ImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n## MobileNetV2Model\n\n[[autodoc]] MobileNetV2Model\n    - forward\n\n## MobileNetV2ForImageClassification\n\n[[autodoc]] MobileNetV2ForImageClassification\n    - forward\n\n## MobileNetV2ForSemanticSegmentation\n\n[[autodoc]] MobileNetV2ForSemanticSegmentation\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: a68ea6bd-093a-47fc-a6c2-786e7a3798ae
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a68ea6bd-093a-47fc-a6c2-786e7a3798ae\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"96800f44-32aa-4d6b-ac46-2f8b81e91026\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bf828b75-f0f7-4f25-a93b-7fde2fa7bae3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aa77a9a75339f6f23f8045526df80b39976f8ffa151274d52e2e3ab690e7c9bb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3782, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "ref_doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "chroma:document": "# MusicGen Melody\n\n## Overview\n\nThe MusicGen Melody model was proposed in [Simple and Controllable Music Generation](https://arxiv.org/abs/2306.05284) by Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi and Alexandre D\u00e9fossez.\n\nMusicGen Melody is a single stage auto-regressive Transformer model capable of generating high-quality music samples conditioned on text descriptions or audio prompts. The text descriptions are passed through a frozen text encoder model to obtain a sequence of hidden-state representations. MusicGen is then trained to predict discrete audio tokens, or *audio codes*, conditioned on these hidden-states. These audio tokens are then decoded using an audio compression model, such as EnCodec, to recover the audio waveform.\n\nThrough an efficient token interleaving pattern, MusicGen does not require a self-supervised semantic representation of the text/audio prompts, thus eliminating the need to cascade multiple models to predict a set of codebooks (e.g. hierarchically or upsampling). Instead, it is able to generate all the codebooks in a single forward pass.\n\nThe abstract from the paper is the following:\n\n*We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen.*\n\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/audiocraft). The pre-trained checkpoints can be found on the [Hugging Face Hub](https://huggingface.co/models?sort=downloads&search=facebook%2Fmusicgen).\n\n\n## Difference with [MusicGen](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen)\n\nThere are two key differences with MusicGen:\n1. The audio prompt is used here as a conditional signal for the generated audio sample, whereas it's used for audio continuation in [MusicGen](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen).\n2. Conditional text and audio signals are concatenated to the decoder's hidden states instead of being used as a cross-attention signal, as in MusicGen.\n\n## Generation\n\nMusicGen Melody is compatible with two generation modes: greedy and sampling. In practice, sampling leads to significantly better results than greedy, thus we encourage sampling mode to be used where possible. Sampling is enabled by default, and can be explicitly specified by setting `do_sample=True` in the call to [`MusicgenMelodyForConditionalGeneration.generate`], or by overriding the model's generation config (see below).\n\nTransformers supports both mono (1-channel) and stereo (2-channel) variants of MusicGen Melody. The mono channel versions generate a single set of codebooks. The stereo versions generate 2 sets of codebooks, 1 for each channel (left/right), and each set of codebooks is decoded independently through the audio compression model. The audio streams for each channel are combined to give the final stereo output."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: bf828b75-f0f7-4f25-a93b-7fde2fa7bae3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bf828b75-f0f7-4f25-a93b-7fde2fa7bae3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"96800f44-32aa-4d6b-ac46-2f8b81e91026\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a68ea6bd-093a-47fc-a6c2-786e7a3798ae\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42a83f0f168f9130fb3cf8dbe92d4227141b300b4253de2bd116d022d17a3b6b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f6e5e34f-2211-4749-be2a-b4175226e2f0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"96c59f2b0ab90fbbff7f02393fa8c59e88b45ab5f4f913fed51f83acf59317b7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2109, \"end_char_idx\": 5012, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "ref_doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "chroma:document": "This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/audiocraft). The pre-trained checkpoints can be found on the [Hugging Face Hub](https://huggingface.co/models?sort=downloads&search=facebook%2Fmusicgen).\n\n\n## Difference with [MusicGen](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen)\n\nThere are two key differences with MusicGen:\n1. The audio prompt is used here as a conditional signal for the generated audio sample, whereas it's used for audio continuation in [MusicGen](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen).\n2. Conditional text and audio signals are concatenated to the decoder's hidden states instead of being used as a cross-attention signal, as in MusicGen.\n\n## Generation\n\nMusicGen Melody is compatible with two generation modes: greedy and sampling. In practice, sampling leads to significantly better results than greedy, thus we encourage sampling mode to be used where possible. Sampling is enabled by default, and can be explicitly specified by setting `do_sample=True` in the call to [`MusicgenMelodyForConditionalGeneration.generate`], or by overriding the model's generation config (see below).\n\nTransformers supports both mono (1-channel) and stereo (2-channel) variants of MusicGen Melody. The mono channel versions generate a single set of codebooks. The stereo versions generate 2 sets of codebooks, 1 for each channel (left/right), and each set of codebooks is decoded independently through the audio compression model. The audio streams for each channel are combined to give the final stereo output.\n\n\n#### Audio Conditional Generation\n\nThe model can generate an audio sample conditioned on a text and an audio prompt through use of the [`MusicgenMelodyProcessor`] to pre-process the inputs.\n\nIn the following examples, we load an audio file using the \ud83e\udd17 Datasets library, which can be pip installed through the command below:\n\n```\npip install --upgrade pip\npip install datasets[audio]\n```\n\nThe audio file we are about to use is loaded as follows:\n```python\n>>> from datasets import load_dataset\n\n>>> dataset = load_dataset(\"sanchit-gandhi/gtzan\", split=\"train\", streaming=True)\n>>> sample = next(iter(dataset))[\"audio\"]\n```\n\nThe audio prompt should ideally be free of the low-frequency signals usually produced by instruments such as drums and bass. The [Demucs](https://github.com/adefossez/demucs/tree/main) model can be used to separate vocals and other signals from the drums and bass components.\n\nIf you wish to use Demucs, you first need to follow the installation steps [here](https://github.com/adefossez/demucs/tree/main?tab=readme-ov-file#for-musicians) before using the following snippet:\n\n```python\nfrom demucs import pretrained\nfrom demucs.apply import apply_model\nfrom demucs.audio import convert_audio\nimport torch"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f6e5e34f-2211-4749-be2a-b4175226e2f0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f6e5e34f-2211-4749-be2a-b4175226e2f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"96800f44-32aa-4d6b-ac46-2f8b81e91026\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bf828b75-f0f7-4f25-a93b-7fde2fa7bae3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af00afebb2b930cdf019f4958c9dc529dbdc4c3762e91e1a1785abde0b679345\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eb8951d1-cb4b-4dc6-99c6-34567f37786a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"440784dd95b26be44a7ceb91f0645fdba070d2a64a7c45b35d9fa45b42d28399\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3785, \"end_char_idx\": 7205, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "ref_doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "chroma:document": "#### Audio Conditional Generation\n\nThe model can generate an audio sample conditioned on a text and an audio prompt through use of the [`MusicgenMelodyProcessor`] to pre-process the inputs.\n\nIn the following examples, we load an audio file using the \ud83e\udd17 Datasets library, which can be pip installed through the command below:\n\n```\npip install --upgrade pip\npip install datasets[audio]\n```\n\nThe audio file we are about to use is loaded as follows:\n```python\n>>> from datasets import load_dataset\n\n>>> dataset = load_dataset(\"sanchit-gandhi/gtzan\", split=\"train\", streaming=True)\n>>> sample = next(iter(dataset))[\"audio\"]\n```\n\nThe audio prompt should ideally be free of the low-frequency signals usually produced by instruments such as drums and bass. The [Demucs](https://github.com/adefossez/demucs/tree/main) model can be used to separate vocals and other signals from the drums and bass components.\n\nIf you wish to use Demucs, you first need to follow the installation steps [here](https://github.com/adefossez/demucs/tree/main?tab=readme-ov-file#for-musicians) before using the following snippet:\n\n```python\nfrom demucs import pretrained\nfrom demucs.apply import apply_model\nfrom demucs.audio import convert_audio\nimport torch\n\n\nwav = torch.tensor(sample[\"array\"]).to(torch.float32)\n\ndemucs = pretrained.get_model('htdemucs')\n\nwav = convert_audio(wav[None], sample[\"sampling_rate\"], demucs.samplerate, demucs.audio_channels)\nwav = apply_model(demucs, wav[None])\n```\n\nYou can then use the following snippet to generate music:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> inputs = processor(\n...     audio=wav,\n...     sampling_rate=demucs.samplerate,\n...     text=[\"80s blues track with groovy saxophone\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nYou can also pass the audio signal directly without using Demucs, although the quality of the generation will probably be degraded:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> inputs = processor(\n...     audio=sample[\"array\"],\n...     sampling_rate=sample[\"sampling_rate\"],\n...     text=[\"80s blues track with groovy saxophone\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe audio outputs are a three-dimensional Torch tensor of shape `(batch_size, num_channels, sequence_length)`. To listen to the generated audio samples, you can either play them in an ipynb notebook:\n\n```python\nfrom IPython.display import Audio\n\nsampling_rate = model.config.audio_encoder.sampling_rate\nAudio(audio_values[0].numpy(), rate=sampling_rate)\n```\n\nOr save them as a `.wav` file using a third-party library, e.g. `soundfile`:\n\n```python\n>>> import soundfile as sf\n\n>>> sampling_rate = model.config.audio_encoder.sampling_rate\n>>> sf.write(\"musicgen_out.wav\", audio_values[0].T.numpy(), sampling_rate)\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: eb8951d1-cb4b-4dc6-99c6-34567f37786a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"eb8951d1-cb4b-4dc6-99c6-34567f37786a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"96800f44-32aa-4d6b-ac46-2f8b81e91026\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6e5e34f-2211-4749-be2a-b4175226e2f0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5a88953e4b7307aea1277c995ab4b4d5f9cbd2bc5451ddb70280a2c516e252f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7d310e3a-5623-4761-a959-9ea431c11426\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"da2379d28eab79230bbbf25d14b76d052dc4a5a4986a3642d75cc65e6d96d3d7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7208, \"end_char_idx\": 8402, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "ref_doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "chroma:document": "### Text-only Conditional Generation\n\nThe same [`MusicgenMelodyProcessor`] can be used to pre-process a text-only prompt. \n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> inputs = processor(\n...     text=[\"80s pop track with bassy drums and synth\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe `guidance_scale` is used in classifier free guidance (CFG), setting the weighting between the conditional logits (which are predicted from the text prompts) and the unconditional logits (which are predicted from an unconditional or 'null' prompt). Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer audio quality. CFG is enabled by setting `guidance_scale > 1`. For best results, use `guidance_scale=3` (default)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7d310e3a-5623-4761-a959-9ea431c11426
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7d310e3a-5623-4761-a959-9ea431c11426\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"96800f44-32aa-4d6b-ac46-2f8b81e91026\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"eb8951d1-cb4b-4dc6-99c6-34567f37786a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"513117411da4754f01ca70df26fe77488f84411a4e654bea8a61a03a7811e5a8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"45293d36-c6b2-4922-bbfe-877518d65545\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f92c813daeb7333dc642c71130faecadc99b02fc3032cdb16a376b9506130403\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7208, \"end_char_idx\": 10846, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "ref_doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "chroma:document": "### Text-only Conditional Generation\n\nThe same [`MusicgenMelodyProcessor`] can be used to pre-process a text-only prompt. \n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> inputs = processor(\n...     text=[\"80s pop track with bassy drums and synth\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\nThe `guidance_scale` is used in classifier free guidance (CFG), setting the weighting between the conditional logits (which are predicted from the text prompts) and the unconditional logits (which are predicted from an unconditional or 'null' prompt). Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer audio quality. CFG is enabled by setting `guidance_scale > 1`. For best results, use `guidance_scale=3` (default).\n\n\nYou can also generate in batch:\n\n```python\n>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration\n>>> from datasets import load_dataset\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/musicgen-melody\")\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> # take the first quarter of the audio sample\n>>> sample_1 = sample[\"array\"][: len(sample[\"array\"]) // 4]\n\n>>> # take the first half of the audio sample\n>>> sample_2 = sample[\"array\"][: len(sample[\"array\"]) // 2]\n\n>>> inputs = processor(\n...     audio=[sample_1, sample_2],\n...     sampling_rate=sample[\"sampling_rate\"],\n...     text=[\"80s blues track with groovy saxophone\", \"90s rock song with loud guitars and heavy drums\"],\n...     padding=True,\n...     return_tensors=\"pt\",\n... )\n>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)\n```\n\n### Unconditional Generation\n\nThe inputs for unconditional (or 'null') generation can be obtained through the method [`MusicgenMelodyProcessor.get_unconditional_inputs`]:\n\n```python\n>>> from transformers import MusicgenMelodyForConditionalGeneration, MusicgenMelodyProcessor\n\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n>>> unconditional_inputs = MusicgenMelodyProcessor.from_pretrained(\"facebook/musicgen-melody\").get_unconditional_inputs(num_samples=1)\n\n>>> audio_values = model.generate(**unconditional_inputs, do_sample=True, max_new_tokens=256)\n```\n\n### Generation Configuration\n\nThe default parameters that control the generation process, such as sampling, guidance scale and number of generated tokens, can be found in the model's generation config, and updated as desired:\n\n```python\n>>> from transformers import MusicgenMelodyForConditionalGeneration\n\n>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\")\n\n>>> # inspect the default generation config\n>>> model.generation_config\n\n>>> # increase the guidance scale to 4.0\n>>> model.generation_config.guidance_scale = 4.0\n\n>>> # decrease the max length to 256 tokens\n>>> model.generation_config.max_length = 256\n```\n\nNote that any arguments passed to the generate method will **supersede** those in the generation config, so setting `do_sample=False` in the call to generate will supersede the setting of `model.generation_config.do_sample` in the generation config."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 45293d36-c6b2-4922-bbfe-877518d65545
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"45293d36-c6b2-4922-bbfe-877518d65545\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"96800f44-32aa-4d6b-ac46-2f8b81e91026\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7d310e3a-5623-4761-a959-9ea431c11426\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8be2275e43c5870899c52108cdfbf69735123ac3b1459fe4fec75ed4be5cdb41\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b79b8938-216c-4485-8593-39e0ffcdfa35\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"89f3b22875cae9e05aa58fe5c5a406219922e37b7353f542321b63cfc2ecd17c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10848, \"end_char_idx\": 13660, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "ref_doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "chroma:document": "## Model Structure\n\nThe MusicGen model can be de-composed into three distinct stages:\n1. Text encoder: maps the text inputs to a sequence of hidden-state representations. The pre-trained MusicGen models use a frozen text encoder from either T5 or Flan-T5.\n2. MusicGen Melody decoder: a language model (LM) that auto-regressively generates audio tokens (or codes) conditional on the encoder hidden-state representations\n3. Audio decoder: used to recover the audio waveform from the audio tokens predicted by the decoder.\n\nThus, the MusicGen model can either be used as a standalone decoder model, corresponding to the class [`MusicgenMelodyForCausalLM`], or as a composite model that includes the text encoder and audio encoder, corresponding to the class [`MusicgenMelodyForConditionalGeneration`]. If only the decoder needs to be loaded from the pre-trained checkpoint, it can be loaded by first specifying the correct config, or be accessed through the `.decoder` attribute of the composite model:\n\n```python\n>>> from transformers import AutoConfig, MusicgenMelodyForCausalLM, MusicgenMelodyForConditionalGeneration\n\n>>> # Option 1: get decoder config and pass to `.from_pretrained`\n>>> decoder_config = AutoConfig.from_pretrained(\"facebook/musicgen-melody\").decoder\n>>> decoder = MusicgenMelodyForCausalLM.from_pretrained(\"facebook/musicgen-melody\", **decoder_config.to_dict())\n\n>>> # Option 2: load the entire composite model, but only return the decoder\n>>> decoder = MusicgenMelodyForConditionalGeneration.from_pretrained(\"facebook/musicgen-melody\").decoder\n```\n\nSince the text encoder and audio encoder models are frozen during training, the MusicGen decoder [`MusicgenMelodyForCausalLM`] can be trained standalone on a dataset of encoder hidden-states and audio codes. For inference, the trained decoder can be combined with the frozen text encoder and audio encoder to recover the composite [`MusicgenMelodyForConditionalGeneration`] model.\n\n## Checkpoint Conversion\n\n- After downloading the original checkpoints from [here](https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md#importing--exporting-models), you can convert them using the **conversion script** available at `src/transformers/models/musicgen_melody/convert_musicgen_melody_transformers.py` with the following command:\n\n```bash\npython src/transformers/models/musicgen_melody/convert_musicgen_melody_transformers.py \\\n    --checkpoint=\"facebook/musicgen-melody\" --pytorch_dump_folder /output/path \n```\n\nTips:\n* MusicGen is trained on the 32kHz checkpoint of Encodec. You should ensure you use a compatible version of the Encodec model.\n* Sampling mode tends to deliver better results than greedy - you can toggle sampling with the variable `do_sample` in the call to [`MusicgenMelodyForConditionalGeneration.generate`]"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b79b8938-216c-4485-8593-39e0ffcdfa35
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/musicgen_melody", "title": "MusicGen Melody", "tokens": 3291, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b79b8938-216c-4485-8593-39e0ffcdfa35\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"96800f44-32aa-4d6b-ac46-2f8b81e91026\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc6ec903118208aefdbdaadd0705201cc1f3ae38c9ce0884545b3456c99b57b7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45293d36-c6b2-4922-bbfe-877518d65545\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/musicgen_melody\", \"title\": \"MusicGen Melody\", \"tokens\": 3291, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cdfb69c847ea5ddb92c9605be10b500094f8b4ed4611a39ce1ce8f06d7d2d4d7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12625, \"end_char_idx\": 14259, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "ref_doc_id": "96800f44-32aa-4d6b-ac46-2f8b81e91026", "chroma:document": "For inference, the trained decoder can be combined with the frozen text encoder and audio encoder to recover the composite [`MusicgenMelodyForConditionalGeneration`] model.\n\n## Checkpoint Conversion\n\n- After downloading the original checkpoints from [here](https://github.com/facebookresearch/audiocraft/blob/main/docs/MUSICGEN.md#importing--exporting-models), you can convert them using the **conversion script** available at `src/transformers/models/musicgen_melody/convert_musicgen_melody_transformers.py` with the following command:\n\n```bash\npython src/transformers/models/musicgen_melody/convert_musicgen_melody_transformers.py \\\n    --checkpoint=\"facebook/musicgen-melody\" --pytorch_dump_folder /output/path \n```\n\nTips:\n* MusicGen is trained on the 32kHz checkpoint of Encodec. You should ensure you use a compatible version of the Encodec model.\n* Sampling mode tends to deliver better results than greedy - you can toggle sampling with the variable `do_sample` in the call to [`MusicgenMelodyForConditionalGeneration.generate`]\n\n\n## MusicgenMelodyDecoderConfig\n\n[[autodoc]] MusicgenMelodyDecoderConfig\n\n## MusicgenMelodyProcessor\n\n[[autodoc]] MusicgenMelodyProcessor\n    - get_unconditional_inputs\n\n## MusicgenMelodyFeatureExtractor\n\n[[autodoc]] MusicgenMelodyFeatureExtractor\n    - _extract_stem_indices\n\n## MusicgenMelodyConfig\n\n[[autodoc]] MusicgenMelodyConfig\n\n## MusicgenMelodyModel\n\n[[autodoc]] MusicgenMelodyModel\n    - forward\n\n## MusicgenMelodyForCausalLM\n\n[[autodoc]] MusicgenMelodyForCausalLM\n    - forward\n\n## MusicgenMelodyForConditionalGeneration\n\n[[autodoc]] MusicgenMelodyForConditionalGeneration\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0d392632-9e64-4b30-9cca-212baa97ae71
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0d392632-9e64-4b30-9cca-212baa97ae71\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9c0a568a-8568-416d-85ca-7d149ce412b9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f8a535a9-ce35-472b-8d16-c360bc2bfea4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d0538b76f38fe4dfddf7af3012bf9511e9aaad69a357a965501d703028a429bf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2925, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "ref_doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "chroma:document": "# GPT-J\n\n## Overview\n\nThe GPT-J model was released in the [kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax) repository by Ben Wang and Aran Komatsuzaki. It is a GPT-2-like\ncausal language model trained on [the Pile](https://pile.eleuther.ai/) dataset.\n\nThis model was contributed by [Stella Biderman](https://huggingface.co/stellaathena).\n\n## Usage tips\n\n- To load [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) in float32 one would need at least 2x model size\n  RAM: 1x for initial weights and another 1x to load the checkpoint. So for GPT-J it would take at least 48GB\n  RAM to just load the model. To reduce the RAM usage there are a few options. The `torch_dtype` argument can be\n  used to initialize the model in half-precision on a CUDA device only. There is also a fp16 branch which stores the fp16 weights,\n  which could be used to further minimize the RAM usage:\n\n```python\n>>> from transformers import GPTJForCausalLM\n>>> import torch\n\n>>> device = \"cuda\"\n>>> model = GPTJForCausalLM.from_pretrained(\n...     \"EleutherAI/gpt-j-6B\",\n...     revision=\"float16\",\n...     torch_dtype=torch.float16,\n... ).to(device)\n```\n\n- The model should fit on 16GB GPU for inference. For training/fine-tuning it would take much more GPU RAM. Adam\n  optimizer for example makes four copies of the model: model, gradients, average and squared average of the gradients.\n  So it would need at least 4x model size GPU memory, even with mixed precision as gradient updates are in fp32. This\n  is not including the activations and data batches, which would again require some more GPU RAM. So one should explore\n  solutions such as DeepSpeed, to train/fine-tune the model. Another option is to use the original codebase to\n  train/fine-tune the model on TPU and then convert the model to Transformers format for inference. Instructions for\n  that could be found [here](https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md)\n\n- Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer. These extra\n  tokens are added for the sake of efficiency on TPUs. To avoid the mismatch between embedding matrix size and vocab\n  size, the tokenizer for [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) contains 143 extra tokens\n  `<|extratoken_1|>... <|extratoken_143|>`, so the `vocab_size` of tokenizer also becomes 50400.\n\n## Usage examples\n\nThe [`~generation.GenerationMixin.generate`] method can be used to generate text using GPT-J\nmodel.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"EleutherAI/gpt-j-6B\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-j-6B\")\n\n>>> prompt = (\n...     \"In a shocking finding, scientists discovered a herd of unicorns living in a remote, \"\n...     \"previously unexplored valley, in the Andes Mountains."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f8a535a9-ce35-472b-8d16-c360bc2bfea4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f8a535a9-ce35-472b-8d16-c360bc2bfea4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9c0a568a-8568-416d-85ca-7d149ce412b9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0d392632-9e64-4b30-9cca-212baa97ae71\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"60a91bc8fe85bb5bfd2ff8b71fdacac99749d46aa2afd985fdd3e38a30dbe600\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"861d5b9d-e5f2-4c2b-9e3c-c4bac71a7749\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8f966e3d8550a2dfc7fb4ce6261a12671387d9e6cd7d5fc7c908a6d7e4bd298a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1403, \"end_char_idx\": 4507, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "ref_doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "chroma:document": "So it would need at least 4x model size GPU memory, even with mixed precision as gradient updates are in fp32. This\n  is not including the activations and data batches, which would again require some more GPU RAM. So one should explore\n  solutions such as DeepSpeed, to train/fine-tune the model. Another option is to use the original codebase to\n  train/fine-tune the model on TPU and then convert the model to Transformers format for inference. Instructions for\n  that could be found [here](https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md)\n\n- Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer. These extra\n  tokens are added for the sake of efficiency on TPUs. To avoid the mismatch between embedding matrix size and vocab\n  size, the tokenizer for [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) contains 143 extra tokens\n  `<|extratoken_1|>... <|extratoken_143|>`, so the `vocab_size` of tokenizer also becomes 50400.\n\n## Usage examples\n\nThe [`~generation.GenerationMixin.generate`] method can be used to generate text using GPT-J\nmodel.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"EleutherAI/gpt-j-6B\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-j-6B\")\n\n>>> prompt = (\n...     \"In a shocking finding, scientists discovered a herd of unicorns living in a remote, \"\n...     \"previously unexplored valley, in the Andes Mountains. Even more surprising to the \"\n...     \"researchers was the fact that the unicorns spoke perfect English.\"\n... )\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n...or in float16 precision:\n\n```python\n>>> from transformers import GPTJForCausalLM, AutoTokenizer\n>>> import torch\n\n>>> device = \"cuda\"\n>>> model = GPTJForCausalLM.from_pretrained(\"EleutherAI/gpt-j-6B\", torch_dtype=torch.float16).to(device)\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-j-6B\")\n\n>>> prompt = (\n...     \"In a shocking finding, scientists discovered a herd of unicorns living in a remote, \"\n...     \"previously unexplored valley, in the Andes Mountains. Even more surprising to the \"\n...     \"researchers was the fact that the unicorns spoke perfect English.\"\n... )\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids.to(device)\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GPT-J. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 861d5b9d-e5f2-4c2b-9e3c-c4bac71a7749
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"861d5b9d-e5f2-4c2b-9e3c-c4bac71a7749\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9c0a568a-8568-416d-85ca-7d149ce412b9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f8a535a9-ce35-472b-8d16-c360bc2bfea4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2fccd3de812277d4c85c6e350da6768f83c3612e59d48f12547ebdf1466beffb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"938e68a5-0aec-4711-85fd-25c51f60525a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bcf573acdfbda20cc3d3718cf1223ecc24decc6444988147c5394faa13b4e153\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2926, \"end_char_idx\": 5529, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "ref_doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "chroma:document": "Even more surprising to the \"\n...     \"researchers was the fact that the unicorns spoke perfect English.\"\n... )\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n...or in float16 precision:\n\n```python\n>>> from transformers import GPTJForCausalLM, AutoTokenizer\n>>> import torch\n\n>>> device = \"cuda\"\n>>> model = GPTJForCausalLM.from_pretrained(\"EleutherAI/gpt-j-6B\", torch_dtype=torch.float16).to(device)\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-j-6B\")\n\n>>> prompt = (\n...     \"In a shocking finding, scientists discovered a herd of unicorns living in a remote, \"\n...     \"previously unexplored valley, in the Andes Mountains. Even more surprising to the \"\n...     \"researchers was the fact that the unicorns spoke perfect English.\"\n... )\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids.to(device)\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GPT-J. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- Description of [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B).\n- A blog on how to [Deploy GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker](https://huggingface.co/blog/gptj-sagemaker).\n- A blog on how to [Accelerate GPT-J inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/gptj-deepspeed-inference).\n- A blog post introducing [GPT-J-6B: 6B JAX-Based Transformer](https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/). \ud83c\udf0e\n- A notebook for [GPT-J-6B Inference Demo](https://colab.research.google.com/github/kingoflolz/mesh-transformer-jax/blob/master/colab_demo.ipynb). \ud83c\udf0e\n- Another notebook demonstrating [Inference with GPT-J-6B](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/GPT-J-6B/Inference_with_GPT_J_6B.ipynb).  \n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 938e68a5-0aec-4711-85fd-25c51f60525a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"938e68a5-0aec-4711-85fd-25c51f60525a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9c0a568a-8568-416d-85ca-7d149ce412b9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"861d5b9d-e5f2-4c2b-9e3c-c4bac71a7749\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"193baca8903a9c58ec827b82b75ba95655021ebe97b161d5bb3b2933342799c5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0dfd8b2d-28de-429e-a75a-521a6341967a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dee81cc5176a66e797e12929df6d4914d2a0edb93de5a43564f76dd7477630c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4281, \"end_char_idx\": 6640, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "ref_doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "chroma:document": "If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- Description of [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B).\n- A blog on how to [Deploy GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker](https://huggingface.co/blog/gptj-sagemaker).\n- A blog on how to [Accelerate GPT-J inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/gptj-deepspeed-inference).\n- A blog post introducing [GPT-J-6B: 6B JAX-Based Transformer](https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/). \ud83c\udf0e\n- A notebook for [GPT-J-6B Inference Demo](https://colab.research.google.com/github/kingoflolz/mesh-transformer-jax/blob/master/colab_demo.ipynb). \ud83c\udf0e\n- Another notebook demonstrating [Inference with GPT-J-6B](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/GPT-J-6B/Inference_with_GPT_J_6B.ipynb).  \n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`GPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0dfd8b2d-28de-429e-a75a-521a6341967a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gptj", "title": "GPT-J", "tokens": 2238, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0dfd8b2d-28de-429e-a75a-521a6341967a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9c0a568a-8568-416d-85ca-7d149ce412b9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"311520b974fb046720566c082be10cd1c1cca473472e3b594267da27bdd01bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"938e68a5-0aec-4711-85fd-25c51f60525a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gptj\", \"title\": \"GPT-J\", \"tokens\": 2238, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87f15ef52b27c83968d5568b6f8cdeeb9c793a1337950874527d88a2d68f27d6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5175, \"end_char_idx\": 7703, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "ref_doc_id": "9c0a568a-8568-416d-85ca-7d149ce412b9", "chroma:document": "\ud83c\udf0e\n- Another notebook demonstrating [Inference with GPT-J-6B](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/GPT-J-6B/Inference_with_GPT_J_6B.ipynb).  \n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`GPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxGPTJForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb).\n\n**Documentation resources**\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## GPTJConfig\n\n[[autodoc]] GPTJConfig\n    - all\n\n<frameworkcontent>\n<pt>\n\n## GPTJModel\n\n[[autodoc]] GPTJModel\n    - forward\n\n## GPTJForCausalLM\n\n[[autodoc]] GPTJForCausalLM\n    - forward\n\n## GPTJForSequenceClassification\n\n[[autodoc]] GPTJForSequenceClassification\n    - forward\n\n## GPTJForQuestionAnswering\n\n[[autodoc]] GPTJForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFGPTJModel\n\n[[autodoc]] TFGPTJModel\n    - call\n\n## TFGPTJForCausalLM\n\n[[autodoc]] TFGPTJForCausalLM\n    - call\n\n## TFGPTJForSequenceClassification\n\n[[autodoc]] TFGPTJForSequenceClassification\n    - call\n\n## TFGPTJForQuestionAnswering\n\n[[autodoc]] TFGPTJForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxGPTJModel\n\n[[autodoc]] FlaxGPTJModel\n    - __call__\n\n## FlaxGPTJForCausalLM\n\n[[autodoc]] FlaxGPTJForCausalLM\n    - __call__\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9ac9a3ef-adfb-4fe3-92d8-c262877ed33e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilevit", "title": "MobileViT", "tokens": 1539, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9ac9a3ef-adfb-4fe3-92d8-c262877ed33e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"39a0d697-900c-4058-80cd-bd5f1e3a0b1b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2a6a7626dcd1e35b5340b766066bbe0dbfa17288134bebbef7f2179372837e8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1141949d-a2e1-4712-afa7-941dd101c9df\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"411cb7705587e026ee8ac6b10a7d1ab7f90c8a09f5a22da18b44060d22afc4f1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3363, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "39a0d697-900c-4058-80cd-bd5f1e3a0b1b", "doc_id": "39a0d697-900c-4058-80cd-bd5f1e3a0b1b", "ref_doc_id": "39a0d697-900c-4058-80cd-bd5f1e3a0b1b", "chroma:document": "# MobileViT\n\n## Overview\n\nThe MobileViT model was proposed in [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://arxiv.org/abs/2110.02178) by Sachin Mehta and Mohammad Rastegari. MobileViT introduces a new layer that replaces local processing in convolutions with global processing using transformers.\n\nThe abstract from the paper is the following:\n\n*Light-weight convolutional neural networks (CNNs) are the de-facto for mobile vision tasks. Their spatial inductive biases allow them to learn representations with fewer parameters across different vision tasks. However, these networks are spatially local. To learn global representations, self-attention-based vision trans-formers (ViTs) have been adopted. Unlike CNNs, ViTs are heavy-weight. In this paper, we ask the following question: is it possible to combine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks? Towards this end, we introduce MobileViT, a light-weight and general-purpose vision transformer for mobile devices. MobileViT presents a different perspective for the global processing of information with transformers, i.e., transformers as convolutions. Our results show that MobileViT significantly outperforms CNN- and ViT-based networks across different tasks and datasets. On the ImageNet-1k dataset, MobileViT achieves top-1 accuracy of 78.4% with about 6 million parameters, which is 3.2% and 6.2% more accurate than MobileNetv3 (CNN-based) and DeIT (ViT-based) for a similar number of parameters. On the MS-COCO object detection task, MobileViT is 5.7% more accurate than MobileNetv3 for a similar number of parameters.*\n\nThis model was contributed by [matthijs](https://huggingface.co/Matthijs). The TensorFlow version of the model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code and weights can be found [here](https://github.com/apple/ml-cvnets).\n\n## Usage tips\n\n- MobileViT is more like a CNN than a Transformer model. It does not work on sequence data but on batches of images. Unlike ViT, there are no embeddings. The backbone model outputs a feature map. You can follow [this tutorial](https://keras.io/examples/vision/mobilevit) for a lightweight introduction.\n- One can use [`MobileViTImageProcessor`] to prepare images for the model. Note that if you do your own preprocessing, the pretrained checkpoints expect images to be in BGR pixel order (not RGB).\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes).\n- The segmentation model uses a [DeepLabV3](https://arxiv.org/abs/1706.05587) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).\n- As the name suggests MobileViT was designed to be performant and efficient on mobile phones. The TensorFlow versions of the MobileViT models are fully compatible with [TensorFlow Lite](https://www.tensorflow.org/lite).\n\n  You can use the following code to convert a MobileViT checkpoint (be it image classification or semantic segmentation) to generate a\n  TensorFlow Lite model:\n\n```py\nfrom transformers import TFMobileViTForImageClassification\nimport tensorflow as tf"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1141949d-a2e1-4712-afa7-941dd101c9df
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilevit", "title": "MobileViT", "tokens": 1539, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1141949d-a2e1-4712-afa7-941dd101c9df\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"39a0d697-900c-4058-80cd-bd5f1e3a0b1b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2a6a7626dcd1e35b5340b766066bbe0dbfa17288134bebbef7f2179372837e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9ac9a3ef-adfb-4fe3-92d8-c262877ed33e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilevit\", \"title\": \"MobileViT\", \"tokens\": 1539, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"091ef4a334f6e3826ddb0bc0e0056a2e737c045e901951bd589952712f4334d9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3366, \"end_char_idx\": 5761, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "39a0d697-900c-4058-80cd-bd5f1e3a0b1b", "doc_id": "39a0d697-900c-4058-80cd-bd5f1e3a0b1b", "ref_doc_id": "39a0d697-900c-4058-80cd-bd5f1e3a0b1b", "chroma:document": "model_ckpt = \"apple/mobilevit-xx-small\"\nmodel = TFMobileViTForImageClassification.from_pretrained(model_ckpt)\n\nconverter = tf.lite.TFLiteConverter.from_keras_model(model)\nconverter.optimizations = [tf.lite.Optimize.DEFAULT]\nconverter.target_spec.supported_ops = [\n    tf.lite.OpsSet.TFLITE_BUILTINS,\n    tf.lite.OpsSet.SELECT_TF_OPS,\n]\ntflite_model = converter.convert()\ntflite_filename = model_ckpt.split(\"/\")[-1] + \".tflite\"\nwith open(tflite_filename, \"wb\") as f:\n    f.write(tflite_model)\n```\n\n  The resulting model will be just **about an MB** making it a good fit for mobile applications where resources and network\n  bandwidth can be constrained.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with MobileViT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`MobileViTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n**Semantic segmentation**\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## MobileViTConfig\n\n[[autodoc]] MobileViTConfig\n\n## MobileViTFeatureExtractor\n\n[[autodoc]] MobileViTFeatureExtractor\n    - __call__\n    - post_process_semantic_segmentation\n\n## MobileViTImageProcessor\n\n[[autodoc]] MobileViTImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n<frameworkcontent>\n<pt>\n\n## MobileViTModel\n\n[[autodoc]] MobileViTModel\n    - forward\n\n## MobileViTForImageClassification\n\n[[autodoc]] MobileViTForImageClassification\n    - forward\n\n## MobileViTForSemanticSegmentation\n\n[[autodoc]] MobileViTForSemanticSegmentation\n    - forward\n\n</pt>\n<tf>\n\n## TFMobileViTModel\n\n[[autodoc]] TFMobileViTModel\n    - call\n\n## TFMobileViTForImageClassification\n\n[[autodoc]] TFMobileViTForImageClassification\n    - call\n\n## TFMobileViTForSemanticSegmentation\n\n[[autodoc]] TFMobileViTForSemanticSegmentation\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1af1dc5e-1893-4977-bc5a-a60a1d284458
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm", "title": "XLM", "tokens": 1480, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1af1dc5e-1893-4977-bc5a-a60a1d284458\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8d947eac-92c8-4dc2-b792-7ec8a3dc6460\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f291245267f20043c181e90db571acc6ad405927b96309c6e5498a4c715d618b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1bc0a7af-8767-4e4a-826c-d8ab336a8810\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ec4aab5bc9c8b14dd3b87b2572c264e59b70ff05301238fd618e8f81b1b4f0d3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3207, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8d947eac-92c8-4dc2-b792-7ec8a3dc6460", "doc_id": "8d947eac-92c8-4dc2-b792-7ec8a3dc6460", "ref_doc_id": "8d947eac-92c8-4dc2-b792-7ec8a3dc6460", "chroma:document": "# XLM\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=xlm\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-xlm-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/xlm-mlm-en-2048\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe XLM model was proposed in [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by\nGuillaume Lample, Alexis Conneau. It's a transformer pretrained using one of the following objectives:\n\n- a causal language modeling (CLM) objective (next token prediction),\n- a masked language modeling (MLM) objective (BERT-like), or\n- a Translation Language Modeling (TLM) object (extension of BERT's MLM to multiple language inputs)\n\nThe abstract from the paper is the following:\n\n*Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding.\nIn this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We\npropose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual\ndata, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain\nstate-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our\napproach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we\nobtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised\nmachine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the\nprevious best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/facebookresearch/XLM/).\n\n## Usage tips\n\n- XLM has many different checkpoints, which were trained using different objectives: CLM, MLM or TLM. Make sure to\n  select the correct objective for your task (e.g. MLM checkpoints are not suitable for generation).\n- XLM has multilingual checkpoints which leverage a specific `lang` parameter. Check out the [multi-lingual](../multilingual) page for more information.\n- A transformer model trained on several languages. There are three different type of training for this model and the library provides checkpoints for all of them:\n\n    * Causal language modeling (CLM) which is the traditional autoregressive training (so this model could be in the previous section as well). One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages.\n    * Masked language modeling (MLM) which is like RoBERTa. One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages, with dynamic masking of the tokens."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1bc0a7af-8767-4e4a-826c-d8ab336a8810
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm", "title": "XLM", "tokens": 1480, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1bc0a7af-8767-4e4a-826c-d8ab336a8810\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8d947eac-92c8-4dc2-b792-7ec8a3dc6460\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f291245267f20043c181e90db571acc6ad405927b96309c6e5498a4c715d618b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1af1dc5e-1893-4977-bc5a-a60a1d284458\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"65c72d0c62e0574d4822cba9827154f9bbf7153df5cb0db5aa40c011e1f288b5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f67aca89-30a1-4bde-aec6-56373dd5698f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7b5675865909b5c551c15eaf437fc64b5042ca674d8d399e543ef31e2f59799\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1537, \"end_char_idx\": 3502, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8d947eac-92c8-4dc2-b792-7ec8a3dc6460", "doc_id": "8d947eac-92c8-4dc2-b792-7ec8a3dc6460", "ref_doc_id": "8d947eac-92c8-4dc2-b792-7ec8a3dc6460", "chroma:document": "On unsupervised machine translation, we\nobtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised\nmachine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the\nprevious best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/facebookresearch/XLM/).\n\n## Usage tips\n\n- XLM has many different checkpoints, which were trained using different objectives: CLM, MLM or TLM. Make sure to\n  select the correct objective for your task (e.g. MLM checkpoints are not suitable for generation).\n- XLM has multilingual checkpoints which leverage a specific `lang` parameter. Check out the [multi-lingual](../multilingual) page for more information.\n- A transformer model trained on several languages. There are three different type of training for this model and the library provides checkpoints for all of them:\n\n    * Causal language modeling (CLM) which is the traditional autoregressive training (so this model could be in the previous section as well). One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages.\n    * Masked language modeling (MLM) which is like RoBERTa. One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages, with dynamic masking of the tokens.\n    * A combination of MLM and translation language modeling (TLM). This consists of concatenating a sentence in two different languages, with random masking. To predict one of the masked tokens, the model can use both, the surrounding context in language 1 and the context given by language 2."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f67aca89-30a1-4bde-aec6-56373dd5698f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm", "title": "XLM", "tokens": 1480, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f67aca89-30a1-4bde-aec6-56373dd5698f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8d947eac-92c8-4dc2-b792-7ec8a3dc6460\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f291245267f20043c181e90db571acc6ad405927b96309c6e5498a4c715d618b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1bc0a7af-8767-4e4a-826c-d8ab336a8810\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm\", \"title\": \"XLM\", \"tokens\": 1480, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b2d05f8044b94ee5af1c607e520797bed61a85a93c84d06d322acaf1be6a320a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1852, \"end_char_idx\": 5279, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8d947eac-92c8-4dc2-b792-7ec8a3dc6460", "doc_id": "8d947eac-92c8-4dc2-b792-7ec8a3dc6460", "ref_doc_id": "8d947eac-92c8-4dc2-b792-7ec8a3dc6460", "chroma:document": "Our code and pretrained models will be made publicly available.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/facebookresearch/XLM/).\n\n## Usage tips\n\n- XLM has many different checkpoints, which were trained using different objectives: CLM, MLM or TLM. Make sure to\n  select the correct objective for your task (e.g. MLM checkpoints are not suitable for generation).\n- XLM has multilingual checkpoints which leverage a specific `lang` parameter. Check out the [multi-lingual](../multilingual) page for more information.\n- A transformer model trained on several languages. There are three different type of training for this model and the library provides checkpoints for all of them:\n\n    * Causal language modeling (CLM) which is the traditional autoregressive training (so this model could be in the previous section as well). One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages.\n    * Masked language modeling (MLM) which is like RoBERTa. One of the languages is selected for each training sample, and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages, with dynamic masking of the tokens.\n    * A combination of MLM and translation language modeling (TLM). This consists of concatenating a sentence in two different languages, with random masking. To predict one of the masked tokens, the model can use both, the surrounding context in language 1 and the context given by language 2.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## XLMConfig\n\n[[autodoc]] XLMConfig\n\n## XLMTokenizer\n\n[[autodoc]] XLMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## XLM specific outputs\n\n[[autodoc]] models.xlm.modeling_xlm.XLMForQuestionAnsweringOutput\n\n<frameworkcontent>\n<pt>\n\n## XLMModel\n\n[[autodoc]] XLMModel\n    - forward\n\n## XLMWithLMHeadModel\n\n[[autodoc]] XLMWithLMHeadModel\n    - forward\n\n## XLMForSequenceClassification\n\n[[autodoc]] XLMForSequenceClassification\n    - forward\n\n## XLMForMultipleChoice\n\n[[autodoc]] XLMForMultipleChoice\n    - forward\n\n## XLMForTokenClassification\n\n[[autodoc]] XLMForTokenClassification\n    - forward\n\n## XLMForQuestionAnsweringSimple\n\n[[autodoc]] XLMForQuestionAnsweringSimple\n    - forward\n\n## XLMForQuestionAnswering\n\n[[autodoc]] XLMForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFXLMModel\n\n[[autodoc]] TFXLMModel\n    - call\n\n## TFXLMWithLMHeadModel\n\n[[autodoc]] TFXLMWithLMHeadModel\n    - call\n\n## TFXLMForSequenceClassification\n\n[[autodoc]] TFXLMForSequenceClassification\n    - call\n\n## TFXLMForMultipleChoice\n\n[[autodoc]] TFXLMForMultipleChoice\n    - call\n\n## TFXLMForTokenClassification\n\n[[autodoc]] TFXLMForTokenClassification\n    - call\n\n## TFXLMForQuestionAnsweringSimple\n\n[[autodoc]] TFXLMForQuestionAnsweringSimple\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: dfbd89cc-d020-4dd8-bb97-22bd60b7f401
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longt5", "title": "LongT5", "tokens": 1472, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dfbd89cc-d020-4dd8-bb97-22bd60b7f401\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14748b87-2a33-4e36-a1c4-8d93b624b3a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"68a76385445061cde8285ffd29f0ca81005c69b8896d504d0a32ed823abfc012\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"180fe7a5-07ec-4646-8813-5a02376af1c2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db10f259194fbc66791fbba6109c540b40916e2a1e7ddd1a60756f52d14a21a4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3414, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14748b87-2a33-4e36-a1c4-8d93b624b3a5", "doc_id": "14748b87-2a33-4e36-a1c4-8d93b624b3a5", "ref_doc_id": "14748b87-2a33-4e36-a1c4-8d93b624b3a5", "chroma:document": "# LongT5\n\n## Overview\n\nThe LongT5 model was proposed in [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://arxiv.org/abs/2112.07916)\nby Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung and Yinfei Yang. It's an\nencoder-decoder transformer pre-trained in a text-to-text denoising generative setting. LongT5 model is an extension of\nT5 model, and it enables using one of the two different efficient attention mechanisms - (1) Local attention, or (2)\nTransient-Global attention.\n\n\nThe abstract from the paper is the following:\n\n*Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the\nperformance of Transformer-based neural models. In this paper, we present a new model, called LongT5, with which we\nexplore the effects of scaling both the input length and model size at the same time. Specifically, we integrated\nattention ideas from long-input transformers (ETC), and adopted pre-training strategies from summarization pre-training\n(PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call {\\em Transient Global}\n(TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are\nable to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on\nquestion answering tasks.*\n\nThis model was contributed by [stancld](https://huggingface.co/stancld).\nThe original code can be found [here](https://github.com/google-research/longt5).\n\n## Usage tips\n\n- [`LongT5ForConditionalGeneration`] is an extension of [`T5ForConditionalGeneration`] exchanging the traditional\nencoder *self-attention* layer with efficient either *local* attention or *transient-global* (*tglobal*) attention.\n- Unlike the T5 model, LongT5 does not use a task prefix. Furthermore, it uses a different pre-training objective\ninspired by the pre-training of [`PegasusForConditionalGeneration`].\n- LongT5 model is designed to work efficiently and very well on long-range *sequence-to-sequence* tasks where the\ninput sequence exceeds commonly used 512 tokens. It is capable of handling input sequences of a length up to 16,384 tokens.\n- For *Local Attention*, the sparse sliding-window local attention operation allows a given token to attend only `r`\ntokens to the left and right of it (with `r=127` by default). *Local Attention* does not introduce any new parameters\nto the model. The complexity of the mechanism is linear in input sequence length `l`: `O(l*r)`.\n- *Transient Global Attention* is an extension of the *Local Attention*. It, furthermore, allows each input token to\ninteract with all other tokens in the layer. This is achieved via splitting an input sequence into blocks of a fixed\nlength `k` (with a default `k=16`). Then, a global token for such a block is obtained via summing and normalizing the embeddings of every token\nin the block. Thanks to this, the attention allows each token to attend to both nearby tokens like in Local attention, and\nalso every global token like in the case of standard global attention (*transient* represents the fact the global tokens\nare constructed dynamically within each attention operation).  As a consequence, *TGlobal* attention introduces\na few new parameters -- global relative position biases and a layer normalization for global token's embedding."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 180fe7a5-07ec-4646-8813-5a02376af1c2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longt5", "title": "LongT5", "tokens": 1472, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"180fe7a5-07ec-4646-8813-5a02376af1c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14748b87-2a33-4e36-a1c4-8d93b624b3a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"68a76385445061cde8285ffd29f0ca81005c69b8896d504d0a32ed823abfc012\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dfbd89cc-d020-4dd8-bb97-22bd60b7f401\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80e65468c7f6f401d2b59fece27c546673541daa50d88752c205e6c5ef6fb26e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f4366e39-3996-4995-afee-122e864428a0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1e22f244a32719f6bc6e71a4017b705709bf670d95814f32e91e277861b9f49f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1817, \"end_char_idx\": 4833, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14748b87-2a33-4e36-a1c4-8d93b624b3a5", "doc_id": "14748b87-2a33-4e36-a1c4-8d93b624b3a5", "ref_doc_id": "14748b87-2a33-4e36-a1c4-8d93b624b3a5", "chroma:document": "- Unlike the T5 model, LongT5 does not use a task prefix. Furthermore, it uses a different pre-training objective\ninspired by the pre-training of [`PegasusForConditionalGeneration`].\n- LongT5 model is designed to work efficiently and very well on long-range *sequence-to-sequence* tasks where the\ninput sequence exceeds commonly used 512 tokens. It is capable of handling input sequences of a length up to 16,384 tokens.\n- For *Local Attention*, the sparse sliding-window local attention operation allows a given token to attend only `r`\ntokens to the left and right of it (with `r=127` by default). *Local Attention* does not introduce any new parameters\nto the model. The complexity of the mechanism is linear in input sequence length `l`: `O(l*r)`.\n- *Transient Global Attention* is an extension of the *Local Attention*. It, furthermore, allows each input token to\ninteract with all other tokens in the layer. This is achieved via splitting an input sequence into blocks of a fixed\nlength `k` (with a default `k=16`). Then, a global token for such a block is obtained via summing and normalizing the embeddings of every token\nin the block. Thanks to this, the attention allows each token to attend to both nearby tokens like in Local attention, and\nalso every global token like in the case of standard global attention (*transient* represents the fact the global tokens\nare constructed dynamically within each attention operation).  As a consequence, *TGlobal* attention introduces\na few new parameters -- global relative position biases and a layer normalization for global token's embedding.\nThe complexity of this mechanism is `O(l(r + l/k))`.\n- An example showing how to evaluate a fine-tuned LongT5 model on the [pubmed dataset](https://huggingface.co/datasets/scientific_papers) is below.\n\n```python\n>>> import evaluate\n>>> from datasets import load_dataset\n>>> from transformers import AutoTokenizer, LongT5ForConditionalGeneration\n\n>>> dataset = load_dataset(\"scientific_papers\", \"pubmed\", split=\"validation\")\n>>> model = (\n...     LongT5ForConditionalGeneration.from_pretrained(\"Stancld/longt5-tglobal-large-16384-pubmed-3k_steps\")\n...     .to(\"cuda\")\n...     .half()\n... )\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Stancld/longt5-tglobal-large-16384-pubmed-3k_steps\")\n\n\n>>> def generate_answers(batch):\n...     inputs_dict = tokenizer(\n...         batch[\"article\"], max_length=16384, padding=\"max_length\", truncation=True, return_tensors=\"pt\"\n...     )\n...     input_ids = inputs_dict.input_ids.to(\"cuda\")\n...     attention_mask = inputs_dict.attention_mask.to(\"cuda\")\n...     output_ids = model.generate(input_ids, attention_mask=attention_mask, max_length=512, num_beams=2)\n...     batch[\"predicted_abstract\"] = tokenizer.batch_decode(output_ids, skip_special_tokens=True)\n...     return batch\n\n\n>>> result = dataset.map(generate_answer, batched=True, batch_size=2)\n>>> rouge = evaluate.load(\"rouge\")\n>>> rouge.compute(predictions=result[\"predicted_abstract\"], references=result[\"abstract\"])\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f4366e39-3996-4995-afee-122e864428a0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longt5", "title": "LongT5", "tokens": 1472, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f4366e39-3996-4995-afee-122e864428a0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14748b87-2a33-4e36-a1c4-8d93b624b3a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"68a76385445061cde8285ffd29f0ca81005c69b8896d504d0a32ed823abfc012\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"180fe7a5-07ec-4646-8813-5a02376af1c2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longt5\", \"title\": \"LongT5\", \"tokens\": 1472, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2d19f14e891c2e1d430221ac0794cf17c50a0a2a5c1b137376f36fd4b1b1b71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3254, \"end_char_idx\": 5495, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14748b87-2a33-4e36-a1c4-8d93b624b3a5", "doc_id": "14748b87-2a33-4e36-a1c4-8d93b624b3a5", "ref_doc_id": "14748b87-2a33-4e36-a1c4-8d93b624b3a5", "chroma:document": "As a consequence, *TGlobal* attention introduces\na few new parameters -- global relative position biases and a layer normalization for global token's embedding.\nThe complexity of this mechanism is `O(l(r + l/k))`.\n- An example showing how to evaluate a fine-tuned LongT5 model on the [pubmed dataset](https://huggingface.co/datasets/scientific_papers) is below.\n\n```python\n>>> import evaluate\n>>> from datasets import load_dataset\n>>> from transformers import AutoTokenizer, LongT5ForConditionalGeneration\n\n>>> dataset = load_dataset(\"scientific_papers\", \"pubmed\", split=\"validation\")\n>>> model = (\n...     LongT5ForConditionalGeneration.from_pretrained(\"Stancld/longt5-tglobal-large-16384-pubmed-3k_steps\")\n...     .to(\"cuda\")\n...     .half()\n... )\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Stancld/longt5-tglobal-large-16384-pubmed-3k_steps\")\n\n\n>>> def generate_answers(batch):\n...     inputs_dict = tokenizer(\n...         batch[\"article\"], max_length=16384, padding=\"max_length\", truncation=True, return_tensors=\"pt\"\n...     )\n...     input_ids = inputs_dict.input_ids.to(\"cuda\")\n...     attention_mask = inputs_dict.attention_mask.to(\"cuda\")\n...     output_ids = model.generate(input_ids, attention_mask=attention_mask, max_length=512, num_beams=2)\n...     batch[\"predicted_abstract\"] = tokenizer.batch_decode(output_ids, skip_special_tokens=True)\n...     return batch\n\n\n>>> result = dataset.map(generate_answer, batched=True, batch_size=2)\n>>> rouge = evaluate.load(\"rouge\")\n>>> rouge.compute(predictions=result[\"predicted_abstract\"], references=result[\"abstract\"])\n```\n\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## LongT5Config\n\n[[autodoc]] LongT5Config\n\n<frameworkcontent>\n<pt>\n\n## LongT5Model\n\n[[autodoc]] LongT5Model\n    - forward\n\n## LongT5ForConditionalGeneration\n\n[[autodoc]] LongT5ForConditionalGeneration\n    - forward\n\n## LongT5EncoderModel\n\n[[autodoc]] LongT5EncoderModel\n    - forward\n\n</pt>\n<jax>\n\n## FlaxLongT5Model\n\n[[autodoc]] FlaxLongT5Model\n    - __call__\n    - encode\n    - decode\n\n## FlaxLongT5ForConditionalGeneration\n\n[[autodoc]] FlaxLongT5ForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 971daf78-0bfd-4662-916d-e0f7e600edc6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cpm", "title": "CPM", "tokens": 670, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"971daf78-0bfd-4662-916d-e0f7e600edc6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cpm\", \"title\": \"CPM\", \"tokens\": 670, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a16e358-6ddf-40f4-9e1e-a45e2f061b25\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cpm\", \"title\": \"CPM\", \"tokens\": 670, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5cd54ffcdd4c1079dbc39fab2cdf38116ea25acd1aaf113469e4a5f846566aa0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1931, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a16e358-6ddf-40f4-9e1e-a45e2f061b25", "doc_id": "7a16e358-6ddf-40f4-9e1e-a45e2f061b25", "ref_doc_id": "7a16e358-6ddf-40f4-9e1e-a45e2f061b25", "chroma:document": "# CPM\n\n## Overview\n\nThe CPM model was proposed in [CPM: A Large-scale Generative Chinese Pre-trained Language Model](https://arxiv.org/abs/2012.00413) by Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin,\nYusheng Su, Haozhe Ji, Jian Guan, Fanchao Qi, Xiaozhi Wang, Yanan Zheng, Guoyang Zeng, Huanqi Cao, Shengqi Chen,\nDaixuan Li, Zhenbo Sun, Zhiyuan Liu, Minlie Huang, Wentao Han, Jie Tang, Juanzi Li, Xiaoyan Zhu, Maosong Sun.\n\nThe abstract from the paper is the following:\n\n*Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3,\nwith 175 billion parameters and 570GB training data, drew a lot of attention due to the capacity of few-shot (even\nzero-shot) learning. However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus\nof GPT-3 is primarily English, and the parameters are not publicly available. In this technical report, we release the\nChinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. To the best\nof our knowledge, CPM, with 2.6 billion parameters and 100GB Chinese training data, is the largest Chinese pre-trained\nlanguage model, which could facilitate several downstream Chinese NLP tasks, such as conversation, essay generation,\ncloze test, and language understanding. Extensive experiments demonstrate that CPM achieves strong performance on many\nNLP tasks in the settings of few-shot (even zero-shot) learning.*\n\nThis model was contributed by [canwenxu](https://huggingface.co/canwenxu). The original implementation can be found\nhere: https://github.com/TsinghuaAI/CPM-Generate\n\n\n<Tip>\n\nCPM's architecture is the same as GPT-2, except for tokenization method. Refer to [GPT-2 documentation](gpt2) for \nAPI reference information.  \n\n</Tip>\n\n\n## CpmTokenizer\n\n[[autodoc]] CpmTokenizer\n\n## CpmTokenizerFast\n\n[[autodoc]] CpmTokenizerFast"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 30671572-3f2c-4a12-a56a-9dd42fc2a315
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/patchtst", "title": "PatchTST", "tokens": 871, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"30671572-3f2c-4a12-a56a-9dd42fc2a315\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtst\", \"title\": \"PatchTST\", \"tokens\": 871, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ec6db4cb-b7a4-4faf-86e6-92b3a1c7bcaa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtst\", \"title\": \"PatchTST\", \"tokens\": 871, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"334974bbfb741d76f27267cb6100117fa0e3c91a81a5ac7dae6971136b4fa8b9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2975, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ec6db4cb-b7a4-4faf-86e6-92b3a1c7bcaa", "doc_id": "ec6db4cb-b7a4-4faf-86e6-92b3a1c7bcaa", "ref_doc_id": "ec6db4cb-b7a4-4faf-86e6-92b3a1c7bcaa", "chroma:document": "# PatchTST\n\n## Overview\n\nThe PatchTST model was proposed in [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://arxiv.org/abs/2211.14730) by Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong and Jayant Kalagnanam.\n\nAt a high level the model vectorizes time series into patches of a given size and encodes the resulting sequence of vectors via a Transformer that then outputs the prediction length forecast via an appropriate head. The model is illustrated in the following figure:\n\n![model](https://github.com/namctin/transformers/assets/8100/150af169-29de-419a-8d98-eb78251c21fa)\n\nThe abstract from the paper is the following:\n\n*We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy.*\n\nThis model was contributed by [namctin](https://huggingface.co/namctin), [gsinthong](https://huggingface.co/gsinthong), [diepi](https://huggingface.co/diepi), [vijaye12](https://huggingface.co/vijaye12), [wmgifford](https://huggingface.co/wmgifford), and [kashif](https://huggingface.co/kashif). The original code can be found [here](https://github.com/yuqinie98/PatchTST).\n\n## Usage tips\n\nThe model can also be used for time series classification and time series regression. See the respective [`PatchTSTForClassification`] and [`PatchTSTForRegression`] classes.\n\n## Resources\n\n- A blog post explaining PatchTST in depth can be found [here](https://huggingface.co/blog/patchtst). The blog can also be opened in Google Colab.\n\n## PatchTSTConfig\n\n[[autodoc]] PatchTSTConfig\n\n## PatchTSTModel\n\n[[autodoc]] PatchTSTModel\n    - forward\n\n## PatchTSTForPrediction\n\n[[autodoc]] PatchTSTForPrediction\n    - forward\n\n## PatchTSTForClassification\n\n[[autodoc]] PatchTSTForClassification\n    - forward\n\n## PatchTSTForPretraining\n\n[[autodoc]] PatchTSTForPretraining\n    - forward\n\n## PatchTSTForRegression\n\n[[autodoc]] PatchTSTForRegression\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b65c17e9-8978-4e91-81d5-103672830f61
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longformer", "title": "Longformer", "tokens": 2001, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b65c17e9-8978-4e91-81d5-103672830f61\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f32c22f1-a782-403a-a305-7ed81d2d223b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1b101f4a3e9c49d0c6a2a27510d3fa148fc242e347775d53d294e12790b22ef3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ea483e2e-5517-468c-8294-78d2481f32c4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"510eec8c6680600eff0cbe37b6e0ef70b2ed66efbc3c6875cbbeb58bac370967\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3103, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "doc_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "ref_doc_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "chroma:document": "# Longformer\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=longformer\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-longformer-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/longformer-base-4096-finetuned-squadv1\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe Longformer model was presented in [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf) by Iz Beltagy, Matthew E. Peters, Arman Cohan.\n\nThe abstract from the paper is the following:\n\n*Transformer-based models are unable to process long sequences due to their self-attention operation, which scales\nquadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention\nmechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or\nlonger. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local\nwindowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we\nevaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In\ncontrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our\npretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on\nWikiHop and TriviaQA.*\n\nThis model was contributed by [beltagy](https://huggingface.co/beltagy). The Authors' code can be found [here](https://github.com/allenai/longformer).\n\n## Usage tips\n\n- Since the Longformer is based on RoBERTa, it doesn't have `token_type_ids`. You don't need to indicate which\n  token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or\n  `</s>`).\n- A transformer model replacing the attention matrices by sparse matrices to go faster. Often, the local context (e.g., what are the two tokens left and right?) is enough to take action for a given token. Some preselected input tokens are still given global attention, but the attention matrix has way less parameters, resulting in a speed-up. See the local attention section for more information.\n\n## Longformer Self Attention\n\nLongformer self attention employs self attention on both a \"local\" context and a \"global\" context. Most tokens only\nattend \"locally\" to each other meaning that each token attends to its \\\\(\\frac{1}{2} w\\\\) previous tokens and\n\\\\(\\frac{1}{2} w\\\\) succeeding tokens with \\\\(w\\\\) being the window length as defined in\n`config.attention_window`. Note that `config.attention_window` can be of type `List` to define a\ndifferent \\\\(w\\\\) for each layer. A selected few tokens attend \"globally\" to all other tokens, as it is\nconventionally done for all tokens in `BertSelfAttention`.\n\nNote that \"locally\" and \"globally\" attending tokens are projected by different query, key and value matrices."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ea483e2e-5517-468c-8294-78d2481f32c4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longformer", "title": "Longformer", "tokens": 2001, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ea483e2e-5517-468c-8294-78d2481f32c4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f32c22f1-a782-403a-a305-7ed81d2d223b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1b101f4a3e9c49d0c6a2a27510d3fa148fc242e347775d53d294e12790b22ef3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b65c17e9-8978-4e91-81d5-103672830f61\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c26ea8b3baed7fa9f8fdb532716ba94d8f8da9f828091dfdaae6ae1cad182b9f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"adac694b-263d-423a-8054-c412616537b6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69c2e8679c53872313ba6f3643743da5e0eb9d58428d6ccf8bb0542fcdbd6cbd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "doc_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "ref_doc_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "chroma:document": "*\n\nThis model was contributed by [beltagy](https://huggingface.co/beltagy). The Authors' code can be found [here](https://github.com/allenai/longformer).\n\n## Usage tips\n\n- Since the Longformer is based on RoBERTa, it doesn't have `token_type_ids`. You don't need to indicate which\n  token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or\n  `</s>`).\n- A transformer model replacing the attention matrices by sparse matrices to go faster. Often, the local context (e.g., what are the two tokens left and right?) is enough to take action for a given token. Some preselected input tokens are still given global attention, but the attention matrix has way less parameters, resulting in a speed-up. See the local attention section for more information.\n\n## Longformer Self Attention\n\nLongformer self attention employs self attention on both a \"local\" context and a \"global\" context. Most tokens only\nattend \"locally\" to each other meaning that each token attends to its \\\\(\\frac{1}{2} w\\\\) previous tokens and\n\\\\(\\frac{1}{2} w\\\\) succeeding tokens with \\\\(w\\\\) being the window length as defined in\n`config.attention_window`. Note that `config.attention_window` can be of type `List` to define a\ndifferent \\\\(w\\\\) for each layer. A selected few tokens attend \"globally\" to all other tokens, as it is\nconventionally done for all tokens in `BertSelfAttention`.\n\nNote that \"locally\" and \"globally\" attending tokens are projected by different query, key and value matrices. Also note\nthat every \"locally\" attending token not only attends to tokens within its window \\\\(w\\\\), but also to all \"globally\"\nattending tokens so that global attention is *symmetric*.\n\nThe user can define which tokens attend \"locally\" and which tokens attend \"globally\" by setting the tensor\n`global_attention_mask` at run-time appropriately. All Longformer models employ the following logic for\n`global_attention_mask`:\n\n- 0: the token attends \"locally\",\n- 1: the token attends \"globally\".\n\nFor more information please also refer to [`~LongformerModel.forward`] method.\n\nUsing Longformer self attention, the memory and time complexity of the query-key matmul operation, which usually\nrepresents the memory and time bottleneck, can be reduced from \\\\(\\mathcal{O}(n_s \\times n_s)\\\\) to\n\\\\(\\mathcal{O}(n_s \\times w)\\\\), with \\\\(n_s\\\\) being the sequence length and \\\\(w\\\\) being the average window\nsize. It is assumed that the number of \"globally\" attending tokens is insignificant as compared to the number of\n\"locally\" attending tokens.\n\nFor more information, please refer to the official [paper](https://arxiv.org/pdf/2004.05150.pdf).\n\n\n## Training\n\n[`LongformerForMaskedLM`] is trained the exact same way [`RobertaForMaskedLM`] is\ntrained and should be used as follows:\n\n```python\ninput_ids = tokenizer.encode(\"This is a sentence from [MASK] training data\", return_tensors=\"pt\")\nmlm_labels = tokenizer.encode(\"This is a sentence from the training data\", return_tensors=\"pt\")\n\nloss = model(input_ids, labels=input_ids, masked_lm_labels=mlm_labels)[0]\n```\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: adac694b-263d-423a-8054-c412616537b6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longformer", "title": "Longformer", "tokens": 2001, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"adac694b-263d-423a-8054-c412616537b6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f32c22f1-a782-403a-a305-7ed81d2d223b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1b101f4a3e9c49d0c6a2a27510d3fa148fc242e347775d53d294e12790b22ef3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ea483e2e-5517-468c-8294-78d2481f32c4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"54fdcd19b3c789bcd315408efff916ee61793ac2220b497614cebc04f479adcb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"434718f0-bf60-41b7-bc50-085ec7447672\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6b9995a05e5b5dbcdd4e82c2e083ecdb8b64923315e0ed5ba31793969c690886\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "doc_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "ref_doc_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "chroma:document": "The user can define which tokens attend \"locally\" and which tokens attend \"globally\" by setting the tensor\n`global_attention_mask` at run-time appropriately. All Longformer models employ the following logic for\n`global_attention_mask`:\n\n- 0: the token attends \"locally\",\n- 1: the token attends \"globally\".\n\nFor more information please also refer to [`~LongformerModel.forward`] method.\n\nUsing Longformer self attention, the memory and time complexity of the query-key matmul operation, which usually\nrepresents the memory and time bottleneck, can be reduced from \\\\(\\mathcal{O}(n_s \\times n_s)\\\\) to\n\\\\(\\mathcal{O}(n_s \\times w)\\\\), with \\\\(n_s\\\\) being the sequence length and \\\\(w\\\\) being the average window\nsize. It is assumed that the number of \"globally\" attending tokens is insignificant as compared to the number of\n\"locally\" attending tokens.\n\nFor more information, please refer to the official [paper](https://arxiv.org/pdf/2004.05150.pdf).\n\n\n## Training\n\n[`LongformerForMaskedLM`] is trained the exact same way [`RobertaForMaskedLM`] is\ntrained and should be used as follows:\n\n```python\ninput_ids = tokenizer.encode(\"This is a sentence from [MASK] training data\", return_tensors=\"pt\")\nmlm_labels = tokenizer.encode(\"This is a sentence from the training data\", return_tensors=\"pt\")\n\nloss = model(input_ids, labels=input_ids, masked_lm_labels=mlm_labels)[0]\n```\n\n## Resources\n\n- [Text classification task guide](./tasks/sequence_classification)\n- [Token classification task guide](./tasks/token_classification)\n- [Question answering task guide](./tasks/question_answering)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## LongformerConfig\n\n[[autodoc]] LongformerConfig\n\n## LongformerTokenizer\n\n[[autodoc]] LongformerTokenizer\n\n## LongformerTokenizerFast\n\n[[autodoc]] LongformerTokenizerFast\n\n## Longformer specific outputs\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerBaseModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerBaseModelOutputWithPooling\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerMaskedLMOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerSequenceClassifierOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerTokenClassifierOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerBaseModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerBaseModelOutputWithPooling\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerMaskedLMOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerQuestionAnsweringModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerSequenceClassifierOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerMultipleChoiceModelOutput\n\n[[autodoc]] models.longformer."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 434718f0-bf60-41b7-bc50-085ec7447672
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/longformer", "title": "Longformer", "tokens": 2001, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"434718f0-bf60-41b7-bc50-085ec7447672\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f32c22f1-a782-403a-a305-7ed81d2d223b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1b101f4a3e9c49d0c6a2a27510d3fa148fc242e347775d53d294e12790b22ef3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"adac694b-263d-423a-8054-c412616537b6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/longformer\", \"title\": \"Longformer\", \"tokens\": 2001, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"060ac3ba1cebc61b0bffdc803d584b06b38385050ac1c298584c375c1971f9ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "doc_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "ref_doc_id": "f32c22f1-a782-403a-a305-7ed81d2d223b", "chroma:document": "/tasks/question_answering)\n- [Masked language modeling task guide](./tasks/masked_language_modeling)\n- [Multiple choice task guide](./tasks/multiple_choice)\n\n## LongformerConfig\n\n[[autodoc]] LongformerConfig\n\n## LongformerTokenizer\n\n[[autodoc]] LongformerTokenizer\n\n## LongformerTokenizerFast\n\n[[autodoc]] LongformerTokenizerFast\n\n## Longformer specific outputs\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerBaseModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerBaseModelOutputWithPooling\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerMaskedLMOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerSequenceClassifierOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput\n\n[[autodoc]] models.longformer.modeling_longformer.LongformerTokenClassifierOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerBaseModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerBaseModelOutputWithPooling\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerMaskedLMOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerQuestionAnsweringModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerSequenceClassifierOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerMultipleChoiceModelOutput\n\n[[autodoc]] models.longformer.modeling_tf_longformer.TFLongformerTokenClassifierOutput\n\n<frameworkcontent>\n<pt>\n\n## LongformerModel\n\n[[autodoc]] LongformerModel\n    - forward\n\n## LongformerForMaskedLM\n\n[[autodoc]] LongformerForMaskedLM\n    - forward\n\n## LongformerForSequenceClassification\n\n[[autodoc]] LongformerForSequenceClassification\n    - forward\n\n## LongformerForMultipleChoice\n\n[[autodoc]] LongformerForMultipleChoice\n    - forward\n\n## LongformerForTokenClassification\n\n[[autodoc]] LongformerForTokenClassification\n    - forward\n\n## LongformerForQuestionAnswering\n\n[[autodoc]] LongformerForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFLongformerModel\n\n[[autodoc]] TFLongformerModel\n    - call\n\n## TFLongformerForMaskedLM\n\n[[autodoc]] TFLongformerForMaskedLM\n    - call\n\n## TFLongformerForQuestionAnswering\n\n[[autodoc]] TFLongformerForQuestionAnswering\n    - call\n\n## TFLongformerForSequenceClassification\n\n[[autodoc]] TFLongformerForSequenceClassification\n    - call\n\n## TFLongformerForTokenClassification\n\n[[autodoc]] TFLongformerForTokenClassification\n    - call\n\n## TFLongformerForMultipleChoice\n\n[[autodoc]] TFLongformerForMultipleChoice\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 874c946e-b7e9-44e8-92d4-d7522b19c55d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/groupvit", "title": "GroupViT", "tokens": 1049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"874c946e-b7e9-44e8-92d4-d7522b19c55d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2a2605f1-a897-48ec-b620-4c3b65856ebb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1872af444c2ff9259be2ed06fa160c8d744ef0df2b30043911cf43c5bd40dfca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50575201-a982-451c-85cb-e4b71989f03f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"febc131d565aa02af5d24c558f1e82b6fbc94ead8c583767174fa405c40855fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2742, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2a2605f1-a897-48ec-b620-4c3b65856ebb", "doc_id": "2a2605f1-a897-48ec-b620-4c3b65856ebb", "ref_doc_id": "2a2605f1-a897-48ec-b620-4c3b65856ebb", "chroma:document": "# GroupViT\n\n## Overview\n\nThe GroupViT model was proposed in [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://arxiv.org/abs/2202.11094) by Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang.\nInspired by [CLIP](clip), GroupViT is a vision-language model that can perform zero-shot semantic segmentation on any given vocabulary categories.\n\nThe abstract from the paper is the following:\n\n*Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision.*\n\nThis model was contributed by [xvjiarui](https://huggingface.co/xvjiarui). The TensorFlow version was contributed by [ariG23498](https://huggingface.co/ariG23498) with the help of [Yih-Dar SHIEH](https://huggingface.co/ydshieh), [Amy Roberts](https://huggingface.co/amyeroberts), and [Joao Gante](https://huggingface.co/joaogante).\nThe original code can be found [here](https://github.com/NVlabs/GroupViT).\n\n## Usage tips\n \n- You may specify `output_segmentation=True` in the forward of `GroupViTModel` to get the segmentation logits of input texts. \n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GroupViT.\n\n- The quickest way to get started with GroupViT is by checking the [example notebooks](https://github.com/xvjiarui/GroupViT/blob/main/demo/GroupViT_hf_inference_notebook.ipynb) (which showcase zero-shot segmentation inference).\n- One can also check out the [HuggingFace Spaces demo](https://huggingface.co/spaces/xvjiarui/GroupViT) to play with GroupViT."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 50575201-a982-451c-85cb-e4b71989f03f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/groupvit", "title": "GroupViT", "tokens": 1049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50575201-a982-451c-85cb-e4b71989f03f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2a2605f1-a897-48ec-b620-4c3b65856ebb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1872af444c2ff9259be2ed06fa160c8d744ef0df2b30043911cf43c5bd40dfca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"874c946e-b7e9-44e8-92d4-d7522b19c55d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/groupvit\", \"title\": \"GroupViT\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"250d81a13e324bb85e404bb486b8872b6e21e7ceccd7bc34d47c990c578cca24\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1235, \"end_char_idx\": 3494, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2a2605f1-a897-48ec-b620-4c3b65856ebb", "doc_id": "2a2605f1-a897-48ec-b620-4c3b65856ebb", "ref_doc_id": "2a2605f1-a897-48ec-b620-4c3b65856ebb", "chroma:document": "With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision.*\n\nThis model was contributed by [xvjiarui](https://huggingface.co/xvjiarui). The TensorFlow version was contributed by [ariG23498](https://huggingface.co/ariG23498) with the help of [Yih-Dar SHIEH](https://huggingface.co/ydshieh), [Amy Roberts](https://huggingface.co/amyeroberts), and [Joao Gante](https://huggingface.co/joaogante).\nThe original code can be found [here](https://github.com/NVlabs/GroupViT).\n\n## Usage tips\n \n- You may specify `output_segmentation=True` in the forward of `GroupViTModel` to get the segmentation logits of input texts. \n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GroupViT.\n\n- The quickest way to get started with GroupViT is by checking the [example notebooks](https://github.com/xvjiarui/GroupViT/blob/main/demo/GroupViT_hf_inference_notebook.ipynb) (which showcase zero-shot segmentation inference).\n- One can also check out the [HuggingFace Spaces demo](https://huggingface.co/spaces/xvjiarui/GroupViT) to play with GroupViT. \n\n## GroupViTConfig\n\n[[autodoc]] GroupViTConfig\n    - from_text_vision_configs\n\n## GroupViTTextConfig\n\n[[autodoc]] GroupViTTextConfig\n\n## GroupViTVisionConfig\n\n[[autodoc]] GroupViTVisionConfig\n\n<frameworkcontent>\n<pt>\n\n## GroupViTModel\n\n[[autodoc]] GroupViTModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## GroupViTTextModel\n\n[[autodoc]] GroupViTTextModel\n    - forward\n\n## GroupViTVisionModel\n\n[[autodoc]] GroupViTVisionModel\n    - forward\n\n</pt>\n<tf>\n\n## TFGroupViTModel\n\n[[autodoc]] TFGroupViTModel\n    - call\n    - get_text_features\n    - get_image_features\n\n## TFGroupViTTextModel\n\n[[autodoc]] TFGroupViTTextModel\n    - call\n\n## TFGroupViTVisionModel\n\n[[autodoc]] TFGroupViTVisionModel\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2ebc0323-bfe3-4324-871c-cf522ff346a6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pix2struct", "title": "Pix2Struct", "tokens": 943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2ebc0323-bfe3-4324-871c-cf522ff346a6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pix2struct\", \"title\": \"Pix2Struct\", \"tokens\": 943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"72b11ec2-2913-45ee-931f-4693f7b30961\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pix2struct\", \"title\": \"Pix2Struct\", \"tokens\": 943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f83a13f234516d0009395133681354697c3014e87f6bc7f3c1e5b6a290811b1c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3455, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "72b11ec2-2913-45ee-931f-4693f7b30961", "doc_id": "72b11ec2-2913-45ee-931f-4693f7b30961", "ref_doc_id": "72b11ec2-2913-45ee-931f-4693f7b30961", "chroma:document": "# Pix2Struct\n\n## Overview\n\nThe Pix2Struct model was proposed in [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://arxiv.org/abs/2210.03347) by Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova.\n\nThe abstract from the paper is the following:\n\n> Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to this diversity, previous work has typically relied on domain-specific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy, we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions are rendered directly on top of the input image. For the first time, we show that a single pretrained model can achieve state-of-the-art results in six out of nine tasks across four domains: documents, illustrations, user interfaces, and natural images.\n\nTips:\n\nPix2Struct has been fine tuned on a variety of tasks and datasets, ranging from image captioning, visual question answering (VQA) over different inputs (books, charts, science diagrams), captioning UI components etc. The full list can be found in Table 1 of the paper.\nWe therefore advise you to use these models for the tasks they have been fine tuned on. For instance, if you want to use Pix2Struct for UI captioning, you should use the model fine tuned on the UI dataset. If you want to use Pix2Struct for image captioning, you should use the model fine tuned on the natural images captioning dataset and so on.\n\nIf you want to use the model to perform conditional text captioning, make sure to use the processor with `add_special_tokens=False`.\n\nThis model was contributed by [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/google-research/pix2struct).\n\n## Resources\n\n- [Fine-tuning Notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb)\n- [All models](https://huggingface.co/models?search=pix2struct)\n\n## Pix2StructConfig\n\n[[autodoc]] Pix2StructConfig\n    - from_text_vision_configs\n\n## Pix2StructTextConfig\n\n[[autodoc]] Pix2StructTextConfig\n\n## Pix2StructVisionConfig\n\n[[autodoc]] Pix2StructVisionConfig\n\n## Pix2StructProcessor\n\n[[autodoc]] Pix2StructProcessor\n\n## Pix2StructImageProcessor\n\n[[autodoc]] Pix2StructImageProcessor\n    - preprocess\n\n## Pix2StructTextModel\n\n[[autodoc]] Pix2StructTextModel\n    - forward\n\n## Pix2StructVisionModel\n\n[[autodoc]] Pix2StructVisionModel\n    - forward\n\n## Pix2StructForConditionalGeneration\n\n[[autodoc]] Pix2StructForConditionalGeneration\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 36eaf999-e5bd-4c65-8151-d2fd3dc15a41
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"36eaf999-e5bd-4c65-8151-d2fd3dc15a41\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"23e5a8be-f8d0-4a33-ad88-796027818c80\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"930bbc40-ded8-4611-8135-ba2175607157\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3fe4070a9d580384c2feca594af49ba9c9af58fb2af77503f1af5dd922fb893c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2096, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "ref_doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "chroma:document": "# AWQ\n\n<Tip>\n\nTry AWQ quantization with this [notebook](https://colab.research.google.com/drive/1HzZH89yAXJaZgwJDhQj9LqSBux932BvY)!\n\n</Tip>\n\n[Activation-aware Weight Quantization (AWQ)](https://hf.co/papers/2306.00978) doesn't quantize all the weights in a model, and instead, it preserves a small percentage of weights that are important for LLM performance. This significantly reduces quantization loss such that you can run models in 4-bit precision without experiencing any performance degradation.\n\nThere are several libraries for quantizing models with the AWQ algorithm, such as [llm-awq](https://github.com/mit-han-lab/llm-awq), [autoawq](https://github.com/casper-hansen/AutoAWQ) or [optimum-intel](https://huggingface.co/docs/optimum/main/en/intel/optimization_inc). Transformers supports loading models quantized with the llm-awq and autoawq libraries. This guide will show you how to load models quantized with autoawq, but the process is similar for llm-awq quantized models.\n\nMake sure you have autoawq installed:\n\n```bash\npip install autoawq\n```\n\nAWQ-quantized models can be identified by checking the `quantization_config` attribute in the model's [config.json](https://huggingface.co/TheBloke/zephyr-7B-alpha-AWQ/blob/main/config.json) file:\n\n```json\n{\n  \"_name_or_path\": \"/workspace/process/huggingfaceh4_zephyr-7b-alpha/source\",\n  \"architectures\": [\n    \"MistralForCausalLM\"\n  ],\n  ...\n  ...\n  ...\n  \"quantization_config\": {\n    \"quant_method\": \"awq\",\n    \"zero_point\": true,\n    \"group_size\": 128,\n    \"bits\": 4,\n    \"version\": \"gemm\"\n  }\n}\n```\n\nA quantized model is loaded with the [`~PreTrainedModel.from_pretrained`] method. If you loaded your model on the CPU, make sure to move it to a GPU device first. Use the `device_map` parameter to specify where to place the model:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"TheBloke/zephyr-7B-alpha-AWQ\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"cuda:0\")\n```\n\nLoading an AWQ-quantized model automatically sets other weights to fp16 by default for performance reasons."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 930bbc40-ded8-4611-8135-ba2175607157
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"930bbc40-ded8-4611-8135-ba2175607157\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"23e5a8be-f8d0-4a33-ad88-796027818c80\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"36eaf999-e5bd-4c65-8151-d2fd3dc15a41\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bb97af195acdca8b43b95e62005ffc3aa85847daaab6154afcfd17ab06f2dece\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c08de54c-af59-4283-a6c7-84b879d06ef5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1558afb1780de96a9ae0fa16c205db5b4514bbe62e1b45e7a233b84f347bf4ca\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 777, \"end_char_idx\": 3709, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "ref_doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "chroma:document": "Transformers supports loading models quantized with the llm-awq and autoawq libraries. This guide will show you how to load models quantized with autoawq, but the process is similar for llm-awq quantized models.\n\nMake sure you have autoawq installed:\n\n```bash\npip install autoawq\n```\n\nAWQ-quantized models can be identified by checking the `quantization_config` attribute in the model's [config.json](https://huggingface.co/TheBloke/zephyr-7B-alpha-AWQ/blob/main/config.json) file:\n\n```json\n{\n  \"_name_or_path\": \"/workspace/process/huggingfaceh4_zephyr-7b-alpha/source\",\n  \"architectures\": [\n    \"MistralForCausalLM\"\n  ],\n  ...\n  ...\n  ...\n  \"quantization_config\": {\n    \"quant_method\": \"awq\",\n    \"zero_point\": true,\n    \"group_size\": 128,\n    \"bits\": 4,\n    \"version\": \"gemm\"\n  }\n}\n```\n\nA quantized model is loaded with the [`~PreTrainedModel.from_pretrained`] method. If you loaded your model on the CPU, make sure to move it to a GPU device first. Use the `device_map` parameter to specify where to place the model:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"TheBloke/zephyr-7B-alpha-AWQ\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"cuda:0\")\n```\n\nLoading an AWQ-quantized model automatically sets other weights to fp16 by default for performance reasons. If you want to load these other weights in a different format, use the `torch_dtype` parameter:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"TheBloke/zephyr-7B-alpha-AWQ\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32)\n```\n\nAWQ quantization can also be combined with [FlashAttention-2](../perf_infer_gpu_one#flashattention-2) to further accelerate inference:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"TheBloke/zephyr-7B-alpha-AWQ\", attn_implementation=\"flash_attention_2\", device_map=\"cuda:0\")\n```\n\n## Fused modules\n\nFused modules offers improved accuracy and performance and it is supported out-of-the-box for AWQ modules for [Llama](https://huggingface.co/meta-llama) and [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) architectures, but you can also fuse AWQ modules for unsupported architectures.\n\n<Tip warning={true}>\n\nFused modules cannot be combined with other optimization techniques such as FlashAttention-2.\n\n</Tip>\n\n<hfoptions id=\"fuse\">\n<hfoption id=\"supported architectures\">\n\nTo enable fused modules for supported architectures, create an [`AwqConfig`] and set the parameters `fuse_max_seq_len` and `do_fuse=True`. The `fuse_max_seq_len` parameter is the total sequence length and it should include the context length and the expected generation length. You can set it to a larger value to be safe.\n\nFor example, to fuse the AWQ modules of the [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c08de54c-af59-4283-a6c7-84b879d06ef5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c08de54c-af59-4283-a6c7-84b879d06ef5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"23e5a8be-f8d0-4a33-ad88-796027818c80\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"930bbc40-ded8-4611-8135-ba2175607157\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b542e8b1930d455227b4368704d4794caf5a296e206ddab4af69ada966098fca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8e71f38d-a7a4-47dd-95ab-a90fd1a2f2fd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e35a2e7ef5dead90955e49fb8f28462a3b7d2bd4eb0839774b28e980fc4e47aa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3055, \"end_char_idx\": 4207, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "ref_doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "chroma:document": "<Tip warning={true}>\n\nFused modules cannot be combined with other optimization techniques such as FlashAttention-2.\n\n</Tip>\n\n<hfoptions id=\"fuse\">\n<hfoption id=\"supported architectures\">\n\nTo enable fused modules for supported architectures, create an [`AwqConfig`] and set the parameters `fuse_max_seq_len` and `do_fuse=True`. The `fuse_max_seq_len` parameter is the total sequence length and it should include the context length and the expected generation length. You can set it to a larger value to be safe.\n\nFor example, to fuse the AWQ modules of the [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model.\n\n```python\nimport torch\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nmodel_id = \"TheBloke/Mistral-7B-OpenOrca-AWQ\"\n\nquantization_config = AwqConfig(\n    bits=4,\n    fuse_max_seq_len=512,\n    do_fuse=True,\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config).to(0)\n```\n\nThe [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model was benchmarked with `batch_size=1` with and without fused modules."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8e71f38d-a7a4-47dd-95ab-a90fd1a2f2fd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8e71f38d-a7a4-47dd-95ab-a90fd1a2f2fd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"23e5a8be-f8d0-4a33-ad88-796027818c80\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c08de54c-af59-4283-a6c7-84b879d06ef5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"982c1d9a6c77db8550c28dc2333f1cf4cb7ebf3e0865c4ca4a0c558fc8041b0a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d1400913-c913-4b26-a5ba-ebcd4e0d90d6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"458962dd4681edb7a1845c0d968b2b9584953dfec1dc96cf6b1ca96313124c10\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3055, \"end_char_idx\": 6567, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "ref_doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "chroma:document": "<Tip warning={true}>\n\nFused modules cannot be combined with other optimization techniques such as FlashAttention-2.\n\n</Tip>\n\n<hfoptions id=\"fuse\">\n<hfoption id=\"supported architectures\">\n\nTo enable fused modules for supported architectures, create an [`AwqConfig`] and set the parameters `fuse_max_seq_len` and `do_fuse=True`. The `fuse_max_seq_len` parameter is the total sequence length and it should include the context length and the expected generation length. You can set it to a larger value to be safe.\n\nFor example, to fuse the AWQ modules of the [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model.\n\n```python\nimport torch\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nmodel_id = \"TheBloke/Mistral-7B-OpenOrca-AWQ\"\n\nquantization_config = AwqConfig(\n    bits=4,\n    fuse_max_seq_len=512,\n    do_fuse=True,\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config).to(0)\n```\n\nThe [TheBloke/Mistral-7B-OpenOrca-AWQ](https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ) model was benchmarked with `batch_size=1` with and without fused modules.\n\n<figcaption class=\"text-center text-gray-500 text-lg\">Unfused module</figcaption>\n\n|   Batch Size |   Prefill Length |   Decode Length |   Prefill tokens/s |   Decode tokens/s | Memory (VRAM)   |\n|-------------:|-----------------:|----------------:|-------------------:|------------------:|:----------------|\n|            1 |               32 |              32 |            60.0984 |           38.4537 | 4.50 GB (5.68%) |\n|            1 |               64 |              64 |          1333.67   |           31.6604 | 4.50 GB (5.68%) |\n|            1 |              128 |             128 |          2434.06   |           31.6272 | 4.50 GB (5.68%) |\n|            1 |              256 |             256 |          3072.26   |           38.1731 | 4.50 GB (5.68%) |\n|            1 |              512 |             512 |          3184.74   |           31.6819 | 4.59 GB (5.80%) |\n|            1 |             1024 |            1024 |          3148.18   |           36.8031 | 4.81 GB (6.07%) |\n|            1 |             2048 |            2048 |          2927.33   |           35.2676 | 5.73 GB (7.23%) |\n\n<figcaption class=\"text-center text-gray-500 text-lg\">Fused module</figcaption>\n\n|   Batch Size |   Prefill Length |   Decode Length |   Prefill tokens/s |   Decode tokens/s | Memory (VRAM)   |\n|-------------:|-----------------:|----------------:|-------------------:|------------------:|:----------------|\n|            1 |               32 |              32 |            81.4899 |           80.2569 | 4.00 GB (5.05%) |\n|            1 |               64 |              64 |          1756.1    |          106.26   | 4.00 GB (5.05%) |\n|            1 |              128 |             128 |          2479.32   |          105.631  | 4.00 GB (5.06%) |\n|            1 |              256 |             256 |          1813.6    |           85.7485 | 4.01 GB (5.06%) |\n|            1 |              512 |             512 |          2848.9    |           97.701  | 4.11 GB (5.19%) |\n|            1 |             1024 |            1024 |          3044.35   |           87.7323 | 4.41 GB (5.57%) |\n|            1 |             2048 |            2048 |          2715.11   |           89.4709 | 5.57 GB (7.04%) |\n\nThe speed and throughput of fused and unfused modules were also tested with the [optimum-benchmark](https://github.com/huggingface/optimum-benchmark) library."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d1400913-c913-4b26-a5ba-ebcd4e0d90d6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d1400913-c913-4b26-a5ba-ebcd4e0d90d6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"23e5a8be-f8d0-4a33-ad88-796027818c80\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8e71f38d-a7a4-47dd-95ab-a90fd1a2f2fd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3d866c4855c64d690c2f6bcc688f05cb27120e0dd3b023bb984ae415eaf90277\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"28be4688-faef-49f1-9581-c760b2d61ae6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"169445644ed42caf1ba5436588d2c8402b6173c0cabb8f8fcd12d55a5845bbbf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6569, \"end_char_idx\": 9291, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "ref_doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "chroma:document": "<div class=\"flex gap-4\">\n  <div>\n    <img class=\"rounded-xl\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/quantization/fused_forward_memory_plot.png\" alt=\"generate throughput per batch size\" />\n    <figcaption class=\"mt-2 text-center text-sm text-gray-500\">forward peak memory/batch size</figcaption>\n  </div>\n  <div>\n    <img class=\"rounded-xl\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/quantization/fused_generate_throughput_plot.png\" alt=\"forward latency per batch size\" />\n    <figcaption class=\"mt-2 text-center text-sm text-gray-500\">generate throughput/batch size</figcaption>\n  </div>\n</div>\n\n</hfoption>\n<hfoption id=\"unsupported architectures\">\n\nFor architectures that don't support fused modules yet, you need to create a custom fusing mapping to define which modules need to be fused with the `modules_to_fuse` parameter. For example, to fuse the AWQ modules of the [TheBloke/Yi-34B-AWQ](https://huggingface.co/TheBloke/Yi-34B-AWQ) model.\n\n```python\nimport torch\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nmodel_id = \"TheBloke/Yi-34B-AWQ\"\n\nquantization_config = AwqConfig(\n    bits=4,\n    fuse_max_seq_len=512,\n    modules_to_fuse={\n        \"attention\": [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n        \"layernorm\": [\"ln1\", \"ln2\", \"norm\"],\n        \"mlp\": [\"gate_proj\", \"up_proj\", \"down_proj\"],\n        \"use_alibi\": False,\n        \"num_attention_heads\": 56,\n        \"num_key_value_heads\": 8,\n        \"hidden_size\": 7168\n    }\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config).to(0)\n```\n\nThe parameter `modules_to_fuse` should include:\n\n- `\"attention\"`: The names of the attention layers to fuse in the following order: query, key, value and output projection layer. If you don't want to fuse these layers, pass an empty list.\n- `\"layernorm\"`: The names of all the LayerNorm layers you want to replace with a custom fused LayerNorm. If you don't want to fuse these layers, pass an empty list.\n- `\"mlp\"`: The names of the MLP layers you want to fuse into a single MLP layer in the order: (gate (dense, layer, post-attention) / up / down layers).\n- `\"use_alibi\"`: If your model uses ALiBi positional embedding.\n- `\"num_attention_heads\"`: The number of attention heads.\n- `\"num_key_value_heads\"`: The number of key value heads that should be used to implement Grouped Query Attention (GQA). If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA), otherwise GQA is used.\n- `\"hidden_size\"`: The dimension of the hidden representations.\n\n</hfoption>\n</hfoptions>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 28be4688-faef-49f1-9581-c760b2d61ae6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"28be4688-faef-49f1-9581-c760b2d61ae6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"23e5a8be-f8d0-4a33-ad88-796027818c80\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d1400913-c913-4b26-a5ba-ebcd4e0d90d6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"68109b7c0849fa382ed3db31723ccf697f71a96b9db06056467165b7c42a6a34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8390, \"end_char_idx\": 10456, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "ref_doc_id": "23e5a8be-f8d0-4a33-ad88-796027818c80", "chroma:document": "If you don't want to fuse these layers, pass an empty list.\n- `\"layernorm\"`: The names of all the LayerNorm layers you want to replace with a custom fused LayerNorm. If you don't want to fuse these layers, pass an empty list.\n- `\"mlp\"`: The names of the MLP layers you want to fuse into a single MLP layer in the order: (gate (dense, layer, post-attention) / up / down layers).\n- `\"use_alibi\"`: If your model uses ALiBi positional embedding.\n- `\"num_attention_heads\"`: The number of attention heads.\n- `\"num_key_value_heads\"`: The number of key value heads that should be used to implement Grouped Query Attention (GQA). If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA), otherwise GQA is used.\n- `\"hidden_size\"`: The dimension of the hidden representations.\n\n</hfoption>\n</hfoptions>\n\n\n\n## ExLlama-v2 support\n\nRecent versions of `autoawq` supports ExLlama-v2 kernels for faster prefill and decoding. To get started, first install the latest version of `autoawq` by running:\n\n```bash\npip install git+https://github.com/casper-hansen/AutoAWQ.git\n```\n\nGet started by passing an `AwqConfig()` with `version=\"exllama\"`.\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, AwqConfig\n\nquantization_config = AwqConfig(version=\"exllama\")\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke/Mistral-7B-Instruct-v0.1-AWQ\",\n    quantization_config=quantization_config,\n    device_map=\"auto\",\n)\n\ninput_ids = torch.randint(0, 100, (1, 128), dtype=torch.long, device=\"cuda\")\noutput = model(input_ids)\nprint(output.logits)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke/Mistral-7B-Instruct-v0.1-AWQ\")\ninput_ids = tokenizer.encode(\"How to make a cake\", return_tensors=\"pt\").to(model.device)\noutput = model.generate(input_ids, do_sample=True, max_length=50, pad_token_id=50256)\nprint(tokenizer.decode(output[0], skip_special_tokens=True))\n```\n\n<Tip warning={true}>\n\nNote this feature is supported on AMD GPUs.\n\n</Tip>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ec8065be-c3a2-4cce-9546-62da3d3ea487
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/eetq", "title": "EETQ", "tokens": 573, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ec8065be-c3a2-4cce-9546-62da3d3ea487\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/eetq\", \"title\": \"EETQ\", \"tokens\": 573, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fa1dc614-711e-454f-a27a-d9a3e34d6d66\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/eetq\", \"title\": \"EETQ\", \"tokens\": 573, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d74a9a8dafc086d595ad90c01f199aafb91cc5c44450e6521d11629dceb1926a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1454, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fa1dc614-711e-454f-a27a-d9a3e34d6d66", "doc_id": "fa1dc614-711e-454f-a27a-d9a3e34d6d66", "ref_doc_id": "fa1dc614-711e-454f-a27a-d9a3e34d6d66", "chroma:document": "# EETQ\n\nThe [EETQ](https://github.com/NetEase-FuXi/EETQ) library supports int8 per-channel weight-only quantization for NVIDIA GPUS. The high-performance GEMM and GEMV kernels are from FasterTransformer and TensorRT-LLM. It requires no calibration dataset and does not need to pre-quantize your model. Moreover, the accuracy degradation is negligible owing to the per-channel quantization. \n\nMake sure you have eetq installed from the [relase page](https://github.com/NetEase-FuXi/EETQ/releases)\n```\npip install --no-cache-dir https://github.com/NetEase-FuXi/EETQ/releases/download/v1.0.0/EETQ-1.0.0+cu121+torch2.1.2-cp310-cp310-linux_x86_64.whl\n```\nor via the source code https://github.com/NetEase-FuXi/EETQ. EETQ requires CUDA capability <= 8.9 and >= 7.0\n```\ngit clone https://github.com/NetEase-FuXi/EETQ.git\ncd EETQ/\ngit submodule update --init --recursive\npip install .\n```\n\nAn unquantized model can be quantized via \"from_pretrained\".\n```py\nfrom transformers import AutoModelForCausalLM, EetqConfig\npath = \"/path/to/model\"\nquantization_config = EetqConfig(\"int8\")\nmodel = AutoModelForCausalLM.from_pretrained(path, device_map=\"auto\", quantization_config=quantization_config)\n```\n\nA quantized model can be saved via \"saved_pretrained\" and be reused again via the \"from_pretrained\".\n\n```py\nquant_path = \"/path/to/save/quantized/model\"\nmodel.save_pretrained(quant_path)\nmodel = AutoModelForCausalLM.from_pretrained(quant_path, device_map=\"auto\")\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: dcb89170-33f8-449f-b9fb-854858488a5e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dcb89170-33f8-449f-b9fb-854858488a5e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c2a0cb4a-aa35-4046-ba66-221a9b8856b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"07fa8ca8-bed9-4fca-9b2b-c175ba905518\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"266f86726e32079555de7a549f73c9095127dbcaa760024b7fc5233d5b49b803\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2518, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "ref_doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "chroma:document": "# bitsandbytes\n\n[bitsandbytes](https://github.com/TimDettmers/bitsandbytes) is the easiest option for quantizing a model to 8 and 4-bit. 8-bit quantization multiplies outliers in fp16 with non-outliers in int8, converts the non-outlier values back to fp16, and then adds them together to return the weights in fp16. This reduces the degradative effect outlier values have on a model's performance. 4-bit quantization compresses a model even further, and it is commonly used with [QLoRA](https://hf.co/papers/2305.14314) to finetune quantized LLMs.\n\nTo use bitsandbytes, make sure you have the following libraries installed:\n\n<hfoptions id=\"bnb\">\n<hfoption id=\"8-bit\">\n\n```bash\npip install transformers accelerate bitsandbytes>0.37.0\n```\n\n</hfoption>\n<hfoption id=\"4-bit\">\n\n```bash\npip install bitsandbytes>=0.39.0\npip install --upgrade accelerate transformers\n```\n\n</hfoption>\n</hfoptions>\n\nNow you can quantize a model by passing a `BitsAndBytesConfig` to [`~PreTrainedModel.from_pretrained`] method. This works for any model in any modality, as long as it supports loading with Accelerate and contains `torch.nn.Linear` layers.\n\n<hfoptions id=\"bnb\">\n<hfoption id=\"8-bit\">\n\nQuantizing a model in 8-bit halves the memory-usage, and for large models, set `device_map=\"auto\"` to efficiently use the GPUs available:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\", \n    quantization_config=quantization_config\n)\n```\n\nBy default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"facebook/opt-350m\", \n    quantization_config=quantization_config, \n    torch_dtype=torch.float32\n)\nmodel_8bit.model.decoder.layers[-1].final_layer_norm.weight.dtype\n```\n\nOnce a model is quantized to 8-bit, you can't push the quantized weights to the Hub unless you're using the latest version of Transformers and bitsandbytes. If you have the latest versions, then you can push the 8-bit model to the Hub with the [`~PreTrainedModel.push_to_hub`] method. The quantization config.json file is pushed first, followed by the quantized model weights."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 07fa8ca8-bed9-4fca-9b2b-c175ba905518
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"07fa8ca8-bed9-4fca-9b2b-c175ba905518\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c2a0cb4a-aa35-4046-ba66-221a9b8856b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dcb89170-33f8-449f-b9fb-854858488a5e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b551169779fb85136adf9528a69abfc464db0a434f72794d725b2a51d04f9258\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"95002e4f-4e7b-4978-b531-f7d70eddef32\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a3fd63bcb35a84fd67fd2d9f9552ef3eabb408a5abc8b7373dc6ceaf9388b653\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1002, \"end_char_idx\": 4010, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "ref_doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "chroma:document": "This works for any model in any modality, as long as it supports loading with Accelerate and contains `torch.nn.Linear` layers.\n\n<hfoptions id=\"bnb\">\n<hfoption id=\"8-bit\">\n\nQuantizing a model in 8-bit halves the memory-usage, and for large models, set `device_map=\"auto\"` to efficiently use the GPUs available:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\", \n    quantization_config=quantization_config\n)\n```\n\nBy default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"facebook/opt-350m\", \n    quantization_config=quantization_config, \n    torch_dtype=torch.float32\n)\nmodel_8bit.model.decoder.layers[-1].final_layer_norm.weight.dtype\n```\n\nOnce a model is quantized to 8-bit, you can't push the quantized weights to the Hub unless you're using the latest version of Transformers and bitsandbytes. If you have the latest versions, then you can push the 8-bit model to the Hub with the [`~PreTrainedModel.push_to_hub`] method. The quantization config.json file is pushed first, followed by the quantized model weights.\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-560m\", \n    quantization_config=quantization_config\n)\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/bloom-560m\")\n\nmodel.push_to_hub(\"bloom-560m-8bit\")\n```\n\n</hfoption>\n<hfoption id=\"4-bit\">\n\nQuantizing a model in 4-bit reduces your memory-usage by 4x, and for large models, set `device_map=\"auto\"` to efficiently use the GPUs available:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_4bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\",\n    quantization_config=quantization_config\n)\n```\n\nBy default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_4bit = AutoModelForCausalLM.from_pretrained(\n    \"facebook/opt-350m\",\n    quantization_config=quantization_config, \n    torch_dtype=torch.float32\n)\nmodel_4bit.model.decoder.layers[-1].final_layer_norm.weight.dtype\n```\n\nIf you have `bitsandbytes>=0.41.3`, you can serialize 4-bit models and push them on Hugging Face Hub."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 95002e4f-4e7b-4978-b531-f7d70eddef32
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"95002e4f-4e7b-4978-b531-f7d70eddef32\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c2a0cb4a-aa35-4046-ba66-221a9b8856b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"07fa8ca8-bed9-4fca-9b2b-c175ba905518\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"81baaec93aa50717899447d49ceeb8117fbc53fd75d07526464d47e0825205c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"16553a1a-209a-4575-8538-baed3cb30ad0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"210688437e42442c50efbb95e26ade4244b2a2b0ab508a713687be81cdcd7660\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2520, \"end_char_idx\": 5338, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "ref_doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "chroma:document": "```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-560m\", \n    quantization_config=quantization_config\n)\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/bloom-560m\")\n\nmodel.push_to_hub(\"bloom-560m-8bit\")\n```\n\n</hfoption>\n<hfoption id=\"4-bit\">\n\nQuantizing a model in 4-bit reduces your memory-usage by 4x, and for large models, set `device_map=\"auto\"` to efficiently use the GPUs available:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_4bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\",\n    quantization_config=quantization_config\n)\n```\n\nBy default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_4bit = AutoModelForCausalLM.from_pretrained(\n    \"facebook/opt-350m\",\n    quantization_config=quantization_config, \n    torch_dtype=torch.float32\n)\nmodel_4bit.model.decoder.layers[-1].final_layer_norm.weight.dtype\n```\n\nIf you have `bitsandbytes>=0.41.3`, you can serialize 4-bit models and push them on Hugging Face Hub. Simply call `model.push_to_hub()` after loading it in 4-bit precision. You can also save the serialized 4-bit models locally with `model.save_pretrained()` command.  \n\n</hfoption>\n</hfoptions>\n\n<Tip warning={true}>\n\nTraining with 8-bit and 4-bit weights are only supported for training *extra* parameters.\n\n</Tip>\n\nYou can check your memory footprint with the `get_memory_footprint` method:\n\n```py\nprint(model.get_memory_footprint())\n```\n\nQuantized models can be loaded from the [`~PreTrainedModel.from_pretrained`] method without needing to specify the `load_in_8bit` or `load_in_4bit` parameters:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/bloom-560m-8bit\", device_map=\"auto\")\n```\n\n## 8-bit (LLM.int8() algorithm)\n\n<Tip>\n\nLearn more about the details of 8-bit quantization in this [blog post](https://huggingface.co/blog/hf-bitsandbytes-integration)!\n\n</Tip>\n\nThis section explores some of the specific features of 8-bit models, such as offloading, outlier thresholds, skipping module conversion, and finetuning.\n\n### Offloading\n\n8-bit models can offload weights between the CPU and GPU to support fitting very large models into memory. The weights dispatched to the CPU are actually stored in **float32**, and aren't converted to 8-bit."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 16553a1a-209a-4575-8538-baed3cb30ad0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"16553a1a-209a-4575-8538-baed3cb30ad0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c2a0cb4a-aa35-4046-ba66-221a9b8856b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"95002e4f-4e7b-4978-b531-f7d70eddef32\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ccffdaa968a8352e3b3b27ff4df3c218bd014c226acfdb11ce1ba8676d7af3a7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"08e847f7-ec4a-4078-8695-8da7ec775fb8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6a927d16b20be179139b6a848267bcad9848d3f412928814c567cd30b5287467\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4011, \"end_char_idx\": 6716, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "ref_doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "chroma:document": "Simply call `model.push_to_hub()` after loading it in 4-bit precision. You can also save the serialized 4-bit models locally with `model.save_pretrained()` command.  \n\n</hfoption>\n</hfoptions>\n\n<Tip warning={true}>\n\nTraining with 8-bit and 4-bit weights are only supported for training *extra* parameters.\n\n</Tip>\n\nYou can check your memory footprint with the `get_memory_footprint` method:\n\n```py\nprint(model.get_memory_footprint())\n```\n\nQuantized models can be loaded from the [`~PreTrainedModel.from_pretrained`] method without needing to specify the `load_in_8bit` or `load_in_4bit` parameters:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/bloom-560m-8bit\", device_map=\"auto\")\n```\n\n## 8-bit (LLM.int8() algorithm)\n\n<Tip>\n\nLearn more about the details of 8-bit quantization in this [blog post](https://huggingface.co/blog/hf-bitsandbytes-integration)!\n\n</Tip>\n\nThis section explores some of the specific features of 8-bit models, such as offloading, outlier thresholds, skipping module conversion, and finetuning.\n\n### Offloading\n\n8-bit models can offload weights between the CPU and GPU to support fitting very large models into memory. The weights dispatched to the CPU are actually stored in **float32**, and aren't converted to 8-bit. For example, to enable offloading for the [bigscience/bloom-1b7](https://huggingface.co/bigscience/bloom-1b7) model, start by creating a [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(llm_int8_enable_fp32_cpu_offload=True)\n```\n\nDesign a custom device map to fit everything on your GPU except for the `lm_head`, which you'll dispatch to the CPU:\n\n```py\ndevice_map = {\n    \"transformer.word_embeddings\": 0,\n    \"transformer.word_embeddings_layernorm\": 0,\n    \"lm_head\": \"cpu\",\n    \"transformer.h\": 0,\n    \"transformer.ln_f\": 0,\n}\n```\n\nNow load your model with the custom `device_map` and `quantization_config`:\n\n```py\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\",\n    device_map=device_map,\n    quantization_config=quantization_config,\n)\n```\n\n### Outlier threshold\n\nAn \"outlier\" is a hidden state value greater than a certain threshold, and these values are computed in fp16. While the values are usually normally distributed ([-3.5, 3.5]), this distribution can be very different for large models ([-60, 6] or [6, 60]). 8-bit quantization works well for values ~5, but beyond that, there is a significant performance penalty. A good default threshold value is 6, but a lower threshold may be needed for more unstable models (small models or finetuning)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 08e847f7-ec4a-4078-8695-8da7ec775fb8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"08e847f7-ec4a-4078-8695-8da7ec775fb8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c2a0cb4a-aa35-4046-ba66-221a9b8856b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"16553a1a-209a-4575-8538-baed3cb30ad0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"86f9284f698a85144003641758f55b67df56b1d7ff7bde20043ff037fa304f50\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"782667b5-b019-4a5b-bb17-8ac9e0316a45\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cba71e72f8eff09b4489101ac527b10c6629832bec214ca7af56b88c69a5b184\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5238, \"end_char_idx\": 8103, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "ref_doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "chroma:document": "The weights dispatched to the CPU are actually stored in **float32**, and aren't converted to 8-bit. For example, to enable offloading for the [bigscience/bloom-1b7](https://huggingface.co/bigscience/bloom-1b7) model, start by creating a [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(llm_int8_enable_fp32_cpu_offload=True)\n```\n\nDesign a custom device map to fit everything on your GPU except for the `lm_head`, which you'll dispatch to the CPU:\n\n```py\ndevice_map = {\n    \"transformer.word_embeddings\": 0,\n    \"transformer.word_embeddings_layernorm\": 0,\n    \"lm_head\": \"cpu\",\n    \"transformer.h\": 0,\n    \"transformer.ln_f\": 0,\n}\n```\n\nNow load your model with the custom `device_map` and `quantization_config`:\n\n```py\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\",\n    device_map=device_map,\n    quantization_config=quantization_config,\n)\n```\n\n### Outlier threshold\n\nAn \"outlier\" is a hidden state value greater than a certain threshold, and these values are computed in fp16. While the values are usually normally distributed ([-3.5, 3.5]), this distribution can be very different for large models ([-60, 6] or [6, 60]). 8-bit quantization works well for values ~5, but beyond that, there is a significant performance penalty. A good default threshold value is 6, but a lower threshold may be needed for more unstable models (small models or finetuning).\n\nTo find the best threshold for your model, we recommend experimenting with the `llm_int8_threshold` parameter in [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nmodel_id = \"bigscience/bloom-1b7\"\n\nquantization_config = BitsAndBytesConfig(\n    llm_int8_threshold=10,\n)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=device_map,\n    quantization_config=quantization_config,\n)\n```\n\n### Skip module conversion\n\nFor some models, like [Jukebox](model_doc/jukebox), you don't need to quantize every module to 8-bit which can actually cause instability. With Jukebox, there are several `lm_head` modules that should be skipped using the `llm_int8_skip_modules` parameter in [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nmodel_id = \"bigscience/bloom-1b7\"\n\nquantization_config = BitsAndBytesConfig(\n    llm_int8_skip_modules=[\"lm_head\"],\n)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=\"auto\",\n    quantization_config=quantization_config,\n)\n```\n\n### Finetuning\n\nWith the [PEFT](https://github.com/huggingface/peft) library, you can finetune large models like [flan-t5-large](https://huggingface.co/google/flan-t5-large) and [facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b) with 8-bit quantization."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 782667b5-b019-4a5b-bb17-8ac9e0316a45
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"782667b5-b019-4a5b-bb17-8ac9e0316a45\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c2a0cb4a-aa35-4046-ba66-221a9b8856b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"08e847f7-ec4a-4078-8695-8da7ec775fb8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"64f94c8389347505f7252aa7a848f0ceafdf7977291077e5b90f33d44cb9003c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ba0919c6-c69e-4118-afd4-363e6f6657be\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a536b22889ac87da0c70b6f9b9995d33bf2d62f286a95bd0c40d6878716fc9e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6589, \"end_char_idx\": 8831, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "ref_doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "chroma:document": "A good default threshold value is 6, but a lower threshold may be needed for more unstable models (small models or finetuning).\n\nTo find the best threshold for your model, we recommend experimenting with the `llm_int8_threshold` parameter in [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nmodel_id = \"bigscience/bloom-1b7\"\n\nquantization_config = BitsAndBytesConfig(\n    llm_int8_threshold=10,\n)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=device_map,\n    quantization_config=quantization_config,\n)\n```\n\n### Skip module conversion\n\nFor some models, like [Jukebox](model_doc/jukebox), you don't need to quantize every module to 8-bit which can actually cause instability. With Jukebox, there are several `lm_head` modules that should be skipped using the `llm_int8_skip_modules` parameter in [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nmodel_id = \"bigscience/bloom-1b7\"\n\nquantization_config = BitsAndBytesConfig(\n    llm_int8_skip_modules=[\"lm_head\"],\n)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=\"auto\",\n    quantization_config=quantization_config,\n)\n```\n\n### Finetuning\n\nWith the [PEFT](https://github.com/huggingface/peft) library, you can finetune large models like [flan-t5-large](https://huggingface.co/google/flan-t5-large) and [facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b) with 8-bit quantization. You don't need to pass the `device_map` parameter for training because it'll automatically load your model on a GPU. However, you can still customize the device map with the `device_map` parameter if you want to (`device_map=\"auto\"` should only be used for inference).\n\n## 4-bit (QLoRA algorithm)\n\n<Tip>\n\nTry 4-bit quantization in this [notebook](https://colab.research.google.com/drive/1ge2F1QSK8Q7h0hn3YKuBCOAS0bK8E0wf) and learn more about it's details in this [blog post](https://huggingface.co/blog/4bit-transformers-bitsandbytes).\n\n</Tip>\n\nThis section explores some of the specific features of 4-bit models, such as changing the compute data type, using the Normal Float 4 (NF4) data type, and using nested quantization."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ba0919c6-c69e-4118-afd4-363e6f6657be
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ba0919c6-c69e-4118-afd4-363e6f6657be\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c2a0cb4a-aa35-4046-ba66-221a9b8856b1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"782667b5-b019-4a5b-bb17-8ac9e0316a45\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"779d5c356a38d788a7894c7d16025dd26139b806da797b84f8ef97c51d69fa34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8104, \"end_char_idx\": 11387, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "ref_doc_id": "c2a0cb4a-aa35-4046-ba66-221a9b8856b1", "chroma:document": "You don't need to pass the `device_map` parameter for training because it'll automatically load your model on a GPU. However, you can still customize the device map with the `device_map` parameter if you want to (`device_map=\"auto\"` should only be used for inference).\n\n## 4-bit (QLoRA algorithm)\n\n<Tip>\n\nTry 4-bit quantization in this [notebook](https://colab.research.google.com/drive/1ge2F1QSK8Q7h0hn3YKuBCOAS0bK8E0wf) and learn more about it's details in this [blog post](https://huggingface.co/blog/4bit-transformers-bitsandbytes).\n\n</Tip>\n\nThis section explores some of the specific features of 4-bit models, such as changing the compute data type, using the Normal Float 4 (NF4) data type, and using nested quantization.\n\n\n### Compute data type\n\nTo speedup computation, you can change the data type from float32 (the default value) to bf16 using the `bnb_4bit_compute_dtype` parameter in [`BitsAndBytesConfig`]:\n\n```py\nimport torch\nfrom transformers import BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)\n```\n\n### Normal Float 4 (NF4)\n\nNF4 is a 4-bit data type from the [QLoRA](https://hf.co/papers/2305.14314) paper, adapted for weights initialized from a normal distribution. You should use NF4 for training 4-bit base models. This can be configured with the `bnb_4bit_quant_type` parameter in the [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import BitsAndBytesConfig\n\nnf4_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n)\n\nmodel_nf4 = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=nf4_config)\n```\n\nFor inference, the `bnb_4bit_quant_type` does not have a huge impact on performance. However, to remain consistent with the model weights, you should use the `bnb_4bit_compute_dtype` and `torch_dtype` values.\n\n### Nested quantization\n\nNested quantization is a technique that can save additional memory at no additional performance cost. This feature performs a second quantization of the already quantized weights to save an addition 0.4 bits/parameter. For example, with nested quantization, you can finetune a [Llama-13b](https://huggingface.co/meta-llama/Llama-2-13b) model on a 16GB NVIDIA T4 GPU with a sequence length of 1024, a batch size of 1, and enabling gradient accumulation with 4 steps.\n\n```py\nfrom transformers import BitsAndBytesConfig\n\ndouble_quant_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_use_double_quant=True,\n)\n\nmodel_double_quant = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-2-13b\", quantization_config=double_quant_config)\n```\n\n## Dequantizing `bitsandbytes` models\n\nOnce quantized, you can dequantize the model to the original precision but this might result in a small quality loss of the model. Make sure you have enough GPU RAM to fit the dequantized model. \n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer\n\nmodel_id = \"facebook/opt-125m\"\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, BitsAndBytesConfig(load_in_4bit=True))\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\nmodel.dequantize()\n\ntext = tokenizer(\"Hello my name is\", return_tensors=\"pt\").to(0)\n\nout = model.generate(**text)\nprint(tokenizer.decode(out[0]))\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 214a865f-ed25-49b7-9739-14623953ae56
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/gptq", "title": "GPTQ", "tokens": 1683, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"214a865f-ed25-49b7-9739-14623953ae56\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1e41434a-354e-4c86-b639-d4cac6f44310\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a479a124df2133faab587f566c0a3c83ef18561238ec1c24a6952fc12a81e8d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73706498-9223-44fb-8453-54ca8600cb5c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0cc1d88ad2a9c0a58df133850f5386334af1047093d0971ebeac45f1bad97f6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2978, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "doc_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "ref_doc_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "chroma:document": "# GPTQ\n\n<Tip>\n\nTry GPTQ quantization with PEFT in this [notebook](https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb?usp=sharing) and learn more about it's details in this [blog post](https://huggingface.co/blog/gptq-integration)!\n\n</Tip>\n\nThe [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) library implements the GPTQ algorithm, a post-training quantization technique where each row of the weight matrix is quantized independently to find a version of the weights that minimizes the error. These weights are quantized to int4, but they're restored to fp16 on the fly during inference. This can save your memory-usage by 4x because the int4 weights are dequantized in a fused kernel rather than a GPU's global memory, and you can also expect a speedup in inference because using a lower bitwidth takes less time to communicate.\n\nBefore you begin, make sure the following libraries are installed:\n\n```bash\npip install auto-gptq\npip install --upgrade accelerate optimum transformers\n```\n\nTo quantize a model (currently only supported for text models), you need to create a [`GPTQConfig`] class and set the number of bits to quantize to, a dataset to calibrate the weights for quantization, and a tokenizer to prepare the dataset.\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig\n\nmodel_id = \"facebook/opt-125m\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\ngptq_config = GPTQConfig(bits=4, dataset=\"c4\", tokenizer=tokenizer)\n```\n\nYou could also pass your own dataset as a list of strings, but it is highly recommended to use the same dataset from the GPTQ paper.\n\n```py\ndataset = [\"auto-gptq is an easy-to-use model quantization library with user-friendly apis, based on GPTQ algorithm.\"]\ngptq_config = GPTQConfig(bits=4, dataset=dataset, tokenizer=tokenizer)\n```\n\nLoad a model to quantize and pass the `gptq_config` to the [`~AutoModelForCausalLM.from_pretrained`] method. Set `device_map=\"auto\"` to automatically offload the model to a CPU to help fit the model in memory, and allow the model modules to be moved between the CPU and GPU for quantization.\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"auto\", quantization_config=gptq_config)\n```\n\nIf you're running out of memory because a dataset is too large, disk offloading is not supported. If this is the case, try passing the `max_memory` parameter to allocate the amount of memory to use on your device (GPU and CPU):\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"auto\", max_memory={0: \"30GiB\", 1: \"46GiB\", \"cpu\": \"30GiB\"}, quantization_config=gptq_config)\n```\n\n<Tip warning={true}>\n\nDepending on your hardware, it can take some time to quantize a model from scratch. It can take ~5 minutes to quantize the [facebook/opt-350m](https://huggingface.co/facebook/opt-350m) model on a free-tier Google Colab GPU, but it'll take ~4 hours to quantize a 175B parameter model on a NVIDIA A100."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 73706498-9223-44fb-8453-54ca8600cb5c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/gptq", "title": "GPTQ", "tokens": 1683, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"73706498-9223-44fb-8453-54ca8600cb5c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1e41434a-354e-4c86-b639-d4cac6f44310\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a479a124df2133faab587f566c0a3c83ef18561238ec1c24a6952fc12a81e8d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"214a865f-ed25-49b7-9739-14623953ae56\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"459e8c30c5a6b0b92fb69565dc838e8a07e0ba1d76ad5d75dd345c7aef72f50c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3438080b-912b-4000-a01f-b722649dcc75\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fbc7ac6e9009608e4090199c0c028b6df1f84552c7d60767fd91a39e8391cc01\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1624, \"end_char_idx\": 4121, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "doc_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "ref_doc_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "chroma:document": "```py\ndataset = [\"auto-gptq is an easy-to-use model quantization library with user-friendly apis, based on GPTQ algorithm.\"]\ngptq_config = GPTQConfig(bits=4, dataset=dataset, tokenizer=tokenizer)\n```\n\nLoad a model to quantize and pass the `gptq_config` to the [`~AutoModelForCausalLM.from_pretrained`] method. Set `device_map=\"auto\"` to automatically offload the model to a CPU to help fit the model in memory, and allow the model modules to be moved between the CPU and GPU for quantization.\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"auto\", quantization_config=gptq_config)\n```\n\nIf you're running out of memory because a dataset is too large, disk offloading is not supported. If this is the case, try passing the `max_memory` parameter to allocate the amount of memory to use on your device (GPU and CPU):\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"auto\", max_memory={0: \"30GiB\", 1: \"46GiB\", \"cpu\": \"30GiB\"}, quantization_config=gptq_config)\n```\n\n<Tip warning={true}>\n\nDepending on your hardware, it can take some time to quantize a model from scratch. It can take ~5 minutes to quantize the [facebook/opt-350m](https://huggingface.co/facebook/opt-350m) model on a free-tier Google Colab GPU, but it'll take ~4 hours to quantize a 175B parameter model on a NVIDIA A100. Before you quantize a model, it is a good idea to check the Hub if a GPTQ-quantized version of the model already exists.\n\n</Tip>\n\nOnce your model is quantized, you can push the model and tokenizer to the Hub where it can be easily shared and accessed. Use the [`~PreTrainedModel.push_to_hub`] method to save the [`GPTQConfig`]:\n\n```py\nquantized_model.push_to_hub(\"opt-125m-gptq\")\ntokenizer.push_to_hub(\"opt-125m-gptq\")\n```\n\nYou could also save your quantized model locally with the [`~PreTrainedModel.save_pretrained`] method. If the model was quantized with the `device_map` parameter, make sure to move the entire model to a GPU or CPU before saving it. For example, to save the model on a CPU:\n\n```py\nquantized_model.save_pretrained(\"opt-125m-gptq\")\ntokenizer.save_pretrained(\"opt-125m-gptq\")\n\n# if quantized with device_map set\nquantized_model.to(\"cpu\")\nquantized_model.save_pretrained(\"opt-125m-gptq\")\n```\n\nReload a quantized model with the [`~PreTrainedModel.from_pretrained`] method, and set `device_map=\"auto\"` to automatically distribute the model on all available GPUs to load the model faster without using more memory than needed."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3438080b-912b-4000-a01f-b722649dcc75
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/gptq", "title": "GPTQ", "tokens": 1683, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3438080b-912b-4000-a01f-b722649dcc75\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1e41434a-354e-4c86-b639-d4cac6f44310\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a479a124df2133faab587f566c0a3c83ef18561238ec1c24a6952fc12a81e8d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73706498-9223-44fb-8453-54ca8600cb5c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8c7108e1a4d5c3cbee68a082ac4c461296b20ff4f9471a4f658df08b62674deb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4c726f15-c5b6-4174-9560-1352a57b7fca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"75cd28e5f9db26468624fcd5dff81c8e48fa5ef452717abc0192fd5b68a74ca5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2762, \"end_char_idx\": 5571, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "doc_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "ref_doc_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "chroma:document": "It can take ~5 minutes to quantize the [facebook/opt-350m](https://huggingface.co/facebook/opt-350m) model on a free-tier Google Colab GPU, but it'll take ~4 hours to quantize a 175B parameter model on a NVIDIA A100. Before you quantize a model, it is a good idea to check the Hub if a GPTQ-quantized version of the model already exists.\n\n</Tip>\n\nOnce your model is quantized, you can push the model and tokenizer to the Hub where it can be easily shared and accessed. Use the [`~PreTrainedModel.push_to_hub`] method to save the [`GPTQConfig`]:\n\n```py\nquantized_model.push_to_hub(\"opt-125m-gptq\")\ntokenizer.push_to_hub(\"opt-125m-gptq\")\n```\n\nYou could also save your quantized model locally with the [`~PreTrainedModel.save_pretrained`] method. If the model was quantized with the `device_map` parameter, make sure to move the entire model to a GPU or CPU before saving it. For example, to save the model on a CPU:\n\n```py\nquantized_model.save_pretrained(\"opt-125m-gptq\")\ntokenizer.save_pretrained(\"opt-125m-gptq\")\n\n# if quantized with device_map set\nquantized_model.to(\"cpu\")\nquantized_model.save_pretrained(\"opt-125m-gptq\")\n```\n\nReload a quantized model with the [`~PreTrainedModel.from_pretrained`] method, and set `device_map=\"auto\"` to automatically distribute the model on all available GPUs to load the model faster without using more memory than needed.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"auto\")\n```\n\n## ExLlama\n\n[ExLlama](https://github.com/turboderp/exllama) is a Python/C++/CUDA implementation of the [Llama](model_doc/llama) model that is designed for faster inference with 4-bit GPTQ weights (check out these [benchmarks](https://github.com/huggingface/optimum/tree/main/tests/benchmark#gptq-benchmark)). The ExLlama kernel is activated by default when you create a [`GPTQConfig`] object. To boost inference speed even further, use the [ExLlamaV2](https://github.com/turboderp/exllamav2) kernels by configuring the `exllama_config` parameter:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, GPTQConfig\n\ngptq_config = GPTQConfig(bits=4, exllama_config={\"version\":2})\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"auto\", quantization_config=gptq_config)\n```\n\n<Tip warning={true}>\n\nOnly 4-bit models are supported, and we recommend deactivating the ExLlama kernels if you're finetuning a quantized model with PEFT.\n\n</Tip>\n\nThe ExLlama kernels are only supported when the entire model is on the GPU. If you're doing inference on a CPU with AutoGPTQ (version > 0.4.2), then you'll need to disable the ExLlama kernel. This overwrites the attributes related to the ExLlama kernels in the quantization config of the config.json file."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4c726f15-c5b6-4174-9560-1352a57b7fca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/gptq", "title": "GPTQ", "tokens": 1683, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4c726f15-c5b6-4174-9560-1352a57b7fca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1e41434a-354e-4c86-b639-d4cac6f44310\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a479a124df2133faab587f566c0a3c83ef18561238ec1c24a6952fc12a81e8d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3438080b-912b-4000-a01f-b722649dcc75\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"49e50f6879e1c3910a618ba4a25a79ff1bce16ceefb13ba3b758c56db8c5fd57\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4123, \"end_char_idx\": 5834, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "doc_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "ref_doc_id": "1e41434a-354e-4c86-b639-d4cac6f44310", "chroma:document": "```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"auto\")\n```\n\n## ExLlama\n\n[ExLlama](https://github.com/turboderp/exllama) is a Python/C++/CUDA implementation of the [Llama](model_doc/llama) model that is designed for faster inference with 4-bit GPTQ weights (check out these [benchmarks](https://github.com/huggingface/optimum/tree/main/tests/benchmark#gptq-benchmark)). The ExLlama kernel is activated by default when you create a [`GPTQConfig`] object. To boost inference speed even further, use the [ExLlamaV2](https://github.com/turboderp/exllamav2) kernels by configuring the `exllama_config` parameter:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, GPTQConfig\n\ngptq_config = GPTQConfig(bits=4, exllama_config={\"version\":2})\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"auto\", quantization_config=gptq_config)\n```\n\n<Tip warning={true}>\n\nOnly 4-bit models are supported, and we recommend deactivating the ExLlama kernels if you're finetuning a quantized model with PEFT.\n\n</Tip>\n\nThe ExLlama kernels are only supported when the entire model is on the GPU. If you're doing inference on a CPU with AutoGPTQ (version > 0.4.2), then you'll need to disable the ExLlama kernel. This overwrites the attributes related to the ExLlama kernels in the quantization config of the config.json file.\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, GPTQConfig\ngptq_config = GPTQConfig(bits=4, use_exllama=False)\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"cpu\", quantization_config=gptq_config)\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5ea3dec0-8651-4df1-8817-1a6c409cff25
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/overview", "title": "Quantization", "tokens": 1348, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5ea3dec0-8651-4df1-8817-1a6c409cff25\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1348, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0baa67bd-df57-4458-bdbb-370a22baf2e7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1348, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"99d71e7aa6433d45660b7b865132a6162014c40365b0db3cc7ed48452b08590d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"48294766-52a0-4b77-b6aa-f3800a065476\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2af429a05642985e45fb0a89dfe9a31bfaba8976d9b3fd0a2ff9e2205baf296d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3515, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0baa67bd-df57-4458-bdbb-370a22baf2e7", "doc_id": "0baa67bd-df57-4458-bdbb-370a22baf2e7", "ref_doc_id": "0baa67bd-df57-4458-bdbb-370a22baf2e7", "chroma:document": "# Quantization\n\nQuantization techniques focus on representing data with less information while also trying to not lose too much accuracy. This often means converting a data type to represent the same information with fewer bits. For example, if your model weights are stored as 32-bit floating points and they're quantized to 16-bit floating points, this halves the model size which makes it easier to store and reduces memory-usage. Lower precision can also speedup inference because it takes less time to perform calculations with fewer bits.\n\n<Tip>\n\nInterested in adding a new quantization method to Transformers? Read the [HfQuantizer](./contribute) guide to learn how!\n\n</Tip>\n\n<Tip>\n\nIf you are new to the quantization field, we recommend you to check out these beginner-friendly courses about quantization in collaboration with DeepLearning.AI:\n\n* [Quantization Fundamentals with Hugging Face](https://www.deeplearning.ai/short-courses/quantization-fundamentals-with-hugging-face/)\n* [Quantization in Depth](https://www.deeplearning.ai/short-courses/quantization-in-depth/)\n\n</Tip>\n\n## When to use what?\n\nThe community has developed many quantization methods for various use cases. With Transformers, you can run any of these integrated methods depending on your use case because each method has their own pros and cons.\n\nFor example, some quantization methods require calibrating the model with a dataset for more accurate and \"extreme\" compression (up to 1-2 bits quantization), while other methods work out of the box with on-the-fly quantization.\n\nAnother parameter to consider is compatibility with your target device. Do you want to quantize on a CPU, GPU, or Apple silicon?\n\nIn short, supporting a wide range of quantization methods allows you to pick the best quantization method for your specific use case.\n\nUse the table below to help you decide which quantization method to use.\n\n| Quantization method                 | On the fly quantization | CPU | CUDA GPU | RoCm GPU (AMD) | Metal (Apple Silicon) | torch.compile() support | Number of bits | Supports fine-tuning (through PEFT) | Serializable with \ud83e\udd17 transformers | \ud83e\udd17 transformers support | Link to library                             |\n|-------------------------------------|-------------------------|-----|----------|----------------|-----------------------|-------------------------|----------------|-------------------------------------|--------------|------------------------|---------------------------------------------|\n| [AQLM](./aqlm)                                | \ud83d\udd34                       |  \ud83d\udfe2   |     \ud83d\udfe2     | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udfe2                      | 1 / 2          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/Vahe1994/AQLM            |\n| [AWQ](./awq) | \ud83d\udd34                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udfe2              | \ud83d\udd34                     | ?                       | 4              | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/casper-hansen/AutoAWQ    |\n| [bitsandbytes](./bitsandbytes)                        | \ud83d\udfe2                       | \ud83d\udd34   |     \ud83d\udfe2     | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udd34                       | 4 / 8          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/TimDettmers/bitsandbytes |\n| [EETQ](./eetq)                                | \ud83d\udfe2                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udd34                     | ?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 48294766-52a0-4b77-b6aa-f3800a065476
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/overview", "title": "Quantization", "tokens": 1348, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"48294766-52a0-4b77-b6aa-f3800a065476\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1348, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0baa67bd-df57-4458-bdbb-370a22baf2e7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1348, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"99d71e7aa6433d45660b7b865132a6162014c40365b0db3cc7ed48452b08590d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5ea3dec0-8651-4df1-8817-1a6c409cff25\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1348, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c053604c41c700a2c63b495bee51681978221c367b6917d212a00f8627de1f20\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1689, \"end_char_idx\": 5245, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0baa67bd-df57-4458-bdbb-370a22baf2e7", "doc_id": "0baa67bd-df57-4458-bdbb-370a22baf2e7", "ref_doc_id": "0baa67bd-df57-4458-bdbb-370a22baf2e7", "chroma:document": "In short, supporting a wide range of quantization methods allows you to pick the best quantization method for your specific use case.\n\nUse the table below to help you decide which quantization method to use.\n\n| Quantization method                 | On the fly quantization | CPU | CUDA GPU | RoCm GPU (AMD) | Metal (Apple Silicon) | torch.compile() support | Number of bits | Supports fine-tuning (through PEFT) | Serializable with \ud83e\udd17 transformers | \ud83e\udd17 transformers support | Link to library                             |\n|-------------------------------------|-------------------------|-----|----------|----------------|-----------------------|-------------------------|----------------|-------------------------------------|--------------|------------------------|---------------------------------------------|\n| [AQLM](./aqlm)                                | \ud83d\udd34                       |  \ud83d\udfe2   |     \ud83d\udfe2     | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udfe2                      | 1 / 2          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/Vahe1994/AQLM            |\n| [AWQ](./awq) | \ud83d\udd34                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udfe2              | \ud83d\udd34                     | ?                       | 4              | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/casper-hansen/AutoAWQ    |\n| [bitsandbytes](./bitsandbytes)                        | \ud83d\udfe2                       | \ud83d\udd34   |     \ud83d\udfe2     | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udd34                       | 4 / 8          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/TimDettmers/bitsandbytes |\n| [EETQ](./eetq)                                | \ud83d\udfe2                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udd34                     | ?                       | 8              | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/NetEase-FuXi/EETQ        |\n| GGUF / GGML (llama.cpp)             | \ud83d\udfe2                       | \ud83d\udfe2   | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udfe2                     | \ud83d\udd34                       | 1 - 8          | \ud83d\udd34                                   | [See GGUF section](../gguf)                | [See GGUF section](../gguf)                      | https://github.com/ggerganov/llama.cpp      |\n| [GPTQ](./gptq)                                | \ud83d\udd34                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udfe2              | \ud83d\udd34                     | \ud83d\udd34                       | 2 - 3 - 4 - 8          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/AutoGPTQ/AutoGPTQ        |\n| [HQQ](./hqq)                                 | \ud83d\udfe2                       | \ud83d\udfe2    | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udfe2                       | 1 - 8          | \ud83d\udfe2                                   | \ud83d\udd34            | \ud83d\udfe2                      | https://github.com/mobiusml/hqq/            |\n| [Quanto](./quanto)                              | \ud83d\udfe2                       | \ud83d\udfe2   | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udfe2                     | \ud83d\udfe2                       | 2 / 4 / 8      | \ud83d\udd34                                   | \ud83d\udd34            | \ud83d\udfe2                      | https://github.com/huggingface/quanto       |\n| [FBGEMM_FP8](./fbgemm_fp8.md)                              | \ud83d\udfe2                       | \ud83d\udd34    | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udd34                      | \ud83d\udd34                        | 8      | \ud83d\udd34                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/pytorch/FBGEMM       |"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7daaae44-cdcc-4352-9f56-ed514a2c93ac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/fbgemm_fp8", "title": "FBGEMM FP8", "tokens": 661, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7daaae44-cdcc-4352-9f56-ed514a2c93ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/fbgemm_fp8\", \"title\": \"FBGEMM FP8\", \"tokens\": 661, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0acb05d0-0e0a-4a7b-a91c-ee22c419fe67\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/fbgemm_fp8\", \"title\": \"FBGEMM FP8\", \"tokens\": 661, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"31eff12f3502d24517b20f40fa7c728cafd487fce4da742da0b93df258622468\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0acb05d0-0e0a-4a7b-a91c-ee22c419fe67", "doc_id": "0acb05d0-0e0a-4a7b-a91c-ee22c419fe67", "ref_doc_id": "0acb05d0-0e0a-4a7b-a91c-ee22c419fe67", "chroma:document": "# FBGEMM FP8\n\nWith FBGEMM FP8 quantization method, you can quantize your model in FP8 (W8A8):\n- the weights will be quantized in 8bit (FP8) per channel\n- the activation will be quantized in 8bit (FP8) per token\n\nIt relies on the [FBGEMM](https://github.com/pytorch/FBGEMM) library which provides efficient low-precision general matrix multiplication for small batch sizes and support for accuracy-loss minimizing techniques such as row-wise quantization and outlier-aware quantization. \n\n> [!TIP]\n> You need a GPU with compute capability>=9 (e.g. H100) \n\nBefore you begin, make sure the following libraries are installed with their latest version:\n\n```bash\npip install --upgrade accelerate fbgemm-gpu torch\n```\n\nIf you are having issues with fbgemm-gpu and torch library, you might need to install the nighlty release. You can follow the instruction [here](https://pytorch.org/FBGEMM/fbgemm_gpu-development/InstallationInstructions.html#fbgemm-gpu-install-libraries:~:text=found%20here.-,Install%20the%20FBGEMM_GPU%20Package,-Install%20through%20PyTorch)\n\n\n```py\nfrom transformers import FbgemmFp8Config, AutoModelForCausalLM, AutoTokenizer\n\nmodel_name = \"meta-llama/Meta-Llama-3-8B\"\nquantization_config = FbgemmFp8Config()\nquantized_model = AutoModelForCausalLM.from_pretrained(model_name, device_map=\"auto\", quantization_config=quantization_config)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\ninput_text = \"What are we having for dinner?\"\ninput_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cuda\")\n\noutput = quantized_model.generate(**input_ids, max_new_tokens=10)\nprint(tokenizer.decode(output[0], skip_special_tokens=True))\n```\n\nA quantized model can be saved via \"saved_pretrained\" and be reused again via the \"from_pretrained\".\n\n```py\nquant_path = \"/path/to/save/quantized/model\"\nmodel.save_pretrained(quant_path)\nmodel = AutoModelForCausalLM.from_pretrained(quant_path, device_map=\"auto\")\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d75d5740-887c-4082-94b6-eb814c7aa3b4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/optimum", "title": "Optimum", "tokens": 242, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d75d5740-887c-4082-94b6-eb814c7aa3b4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/optimum\", \"title\": \"Optimum\", \"tokens\": 242, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f114e5cf-4617-40a4-8a14-decfa0d7ebbe\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/optimum\", \"title\": \"Optimum\", \"tokens\": 242, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"92fb8e5a7999696bb2600dfc4a76f075bb402d115c7ea5e374469945cbb64e85\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 345, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f114e5cf-4617-40a4-8a14-decfa0d7ebbe", "doc_id": "f114e5cf-4617-40a4-8a14-decfa0d7ebbe", "ref_doc_id": "f114e5cf-4617-40a4-8a14-decfa0d7ebbe", "chroma:document": "# Optimum\n\nThe [Optimum](https://huggingface.co/docs/optimum/index) library supports quantization for Intel, Furiosa, ONNX Runtime, GPTQ, and lower-level PyTorch quantization functions. Consider using Optimum for quantization if you're using specific and optimized hardware like Intel CPUs, Furiosa NPUs or a model accelerator like ONNX Runtime."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f1eb8a4f-9c42-4675-96fe-a5cba88c0f4e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/quanto", "title": "Quanto", "tokens": 794, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f1eb8a4f-9c42-4675-96fe-a5cba88c0f4e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/quanto\", \"title\": \"Quanto\", \"tokens\": 794, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c6de8420-b41c-4be5-8263-0a9b8db8a488\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/quanto\", \"title\": \"Quanto\", \"tokens\": 794, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec0c0e135462c2ee65b22053c3ce8571cf03c81eb523923823127e9efb031a89\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2583, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c6de8420-b41c-4be5-8263-0a9b8db8a488", "doc_id": "c6de8420-b41c-4be5-8263-0a9b8db8a488", "ref_doc_id": "c6de8420-b41c-4be5-8263-0a9b8db8a488", "chroma:document": "# Quanto\n\n<Tip>\n\nTry Quanto + transformers with this [notebook](https://colab.research.google.com/drive/16CXfVmtdQvciSh9BopZUDYcmXCDpvgrT?usp=sharing)!\n\n</Tip>\n\n\n[\ud83e\udd17 Quanto](https://github.com/huggingface/quanto) library is a versatile pytorch quantization toolkit. The quantization method used is the linear quantization. Quanto provides several unique features such as:\n\n- weights quantization (`float8`,`int8`,`int4`,`int2`)\n- activation quantization (`float8`,`int8`)\n- modality agnostic (e.g CV,LLM)\n- device agnostic (e.g CUDA,MPS,CPU)\n- compatibility with `torch.compile`\n- easy to add custom kernel for specific device\n- supports quantization aware training\n<!-- Add link to the blogpost -->\n\nBefore you begin, make sure the following libraries are installed:\n\n```bash\npip install quanto accelerate transformers\n```\n\nNow you can quantize a model by passing [`QuantoConfig`] object in the [`~PreTrainedModel.from_pretrained`] method. This works for any model in any modality, as long as it contains `torch.nn.Linear` layers. \n\nThe integration with transformers only supports weights quantization. For the more complex use case such as activation quantization, calibration and quantization aware training, you should use [quanto](https://github.com/huggingface/quanto) library instead. \n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, QuantoConfig\n\nmodel_id = \"facebook/opt-125m\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nquantization_config = QuantoConfig(weights=\"int8\")\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"cuda:0\", quantization_config=quantization_config)\n```\n\nNote that serialization is not supported yet with transformers but it is coming soon! If you want to save the model, you can use quanto library instead.\n\nQuanto library uses linear quantization algorithm for quantization. Even though this is a basic quantization technique, we get very good results! Have a look at the following becnhmark (llama-2-7b on perplexity metric). You can find more benchamarks [here](https://github.com/huggingface/quanto/tree/main/bench/generation)\n\n<div class=\"flex gap-4\">\n  <div>\n    <img class=\"rounded-xl\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/quantization/NousResearch-Llama-2-7b-hf_Perplexity.png\" alt=\"llama-2-7b-quanto-perplexity\" />\n  </div>\n</div>\n\nThe library is versatible enough to be compatible with most PTQ optimization algorithms. The plan in the future is to integrate the most popular algorithms in the most seamless possible way (AWQ, Smoothquant)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 068c9fe4-3616-42e4-a836-0f30a4fbf153
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/contribute", "title": "Contribute new quantization method", "tokens": 1705, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"068c9fe4-3616-42e4-a836-0f30a4fbf153\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5080c521-e9b9-44e6-951f-5474f9191f04\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53fbf61f9596f0cbe6ceca6e165cb2aa79c2308dae7bcd4afac9e225c76e7e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7dfba02f-86bd-40d8-ba19-c6a7725ec84e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"923c92d9fac385259515586e3549ce9638141d1d78921a2c3ce003e5c3e81172\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3192, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "doc_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "ref_doc_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "chroma:document": "# Contribute new quantization method\n\nTransformers supports and integrates many quantization methods such as QLoRA, GPTQ, LLM.int8, and AWQ. However, there are other quantization approaches that are not yet integrated. To make adding and using these quantization methods with Transformers models easier, you should use the [`HfQuantizer`] class. The [`HfQuantizer`] is designed as an internal helper class for adding a quantization method instead of something you apply to every PyTorch module.\n\nThis guide will show you how to integrate a new quantization method with the [`HfQuantizer`] class.\n\n## Requirements\n\nBefore integrating a new quantization method into Transformers, ensure the method you are trying to add meets the following prerequisites. Only quantization methods that can be run with PyTorch modules are currently supported.\n\n- The quantization method is available through a Python package that is pip-installable by anyone (it is also fine if you can only install the package from source). Ideally, pre-compiled kernels are included in the pip package.\n- The method can run on commonly-used hardware (CPU, GPU, ...).\n- The method is wrapped in a `nn.Module` (e.g., `Linear8bitLt`, `Linear4bit`), and the quantized linear layer should have the following definition:\n\n```py\nclass Linear4bit(nn.Module):\n    def __init__(self, ...):\n        ...\n    \n    def forward(self, x):\n        return my_4bit_kernel(x, self.weight, self.bias)\n```\n\nThis way, Transformers models can be easily quantized by replacing some instances of `nn.Linear` with a target class.\n\n- The quantization method should be serializable. You can save the quantized weights locally or push them to the Hub.\n- Make sure the package that contains the quantization kernels/primitive is stable (no frequent breaking changes).\n\nFor some quantization methods, they may require \"pre-quantizing\" the models through data calibration (e.g., AWQ). In this case, we prefer to only support inference in Transformers and let the third-party library maintained by the ML community deal with the model quantization itself.\n\n## Build a new HFQuantizer class\n\n1. Create a new quantization config class inside [src/transformers/utils/quantization_config.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/utils/quantization_config.py) and make sure to expose the new quantization config inside Transformers main `init` by adding it to the [`_import_structure`](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/__init__.py#L1088) object of [src/transformers/__init__.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/__init__.py).\n\n2. Create a new file inside [src/transformers/quantizers/](https://github.com/huggingface/transformers/tree/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers) named `quantizer_your_method.py`, and make it inherit from [src/transformers/quantizers/base.py::HfQuantizer](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/base.py#L28)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7dfba02f-86bd-40d8-ba19-c6a7725ec84e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/contribute", "title": "Contribute new quantization method", "tokens": 1705, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7dfba02f-86bd-40d8-ba19-c6a7725ec84e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5080c521-e9b9-44e6-951f-5474f9191f04\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53fbf61f9596f0cbe6ceca6e165cb2aa79c2308dae7bcd4afac9e225c76e7e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"068c9fe4-3616-42e4-a836-0f30a4fbf153\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b56c88b85a7075ba1efb1648cb6feb05210f57ce70218cbd7c91aa08f32d558a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"625437f6-95fd-4c97-9378-2f370c9db7c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"17ea5e59814c4736a2335e1e30bee769f99f0b0618b8fc13165b59164eb7f821\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1805, \"end_char_idx\": 4854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "doc_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "ref_doc_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "chroma:document": "For some quantization methods, they may require \"pre-quantizing\" the models through data calibration (e.g., AWQ). In this case, we prefer to only support inference in Transformers and let the third-party library maintained by the ML community deal with the model quantization itself.\n\n## Build a new HFQuantizer class\n\n1. Create a new quantization config class inside [src/transformers/utils/quantization_config.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/utils/quantization_config.py) and make sure to expose the new quantization config inside Transformers main `init` by adding it to the [`_import_structure`](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/__init__.py#L1088) object of [src/transformers/__init__.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/__init__.py).\n\n2. Create a new file inside [src/transformers/quantizers/](https://github.com/huggingface/transformers/tree/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers) named `quantizer_your_method.py`, and make it inherit from [src/transformers/quantizers/base.py::HfQuantizer](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/base.py#L28). Make sure to add the new quantizer and quantization config in the quantization auto-mapping in [src/transformers/quantizers/auto.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/auto.py).\n\n3. Define the following class attributes/property methods for your quantization method:\n\n* `requires_calibration`: Whether the quantization method requires a data calibration process. If set to `True`, you can only support inference (with quantized weights) and not inference and quantization.\n* `required_packages`: A list of strings of the required packages to use the quantized weights. You might need to define some new utility methods such as `is_auto_awq_available` in [transformers/src/utils/import_utils.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/utils/import_utils.py).\n* `requires_parameters_quantization`: Only required if your quantization method requires extra attention to the underlying `nn.Parameter` object. For example, bitsandbytes uses `Params4bit` and `Int8Param`, which requires some extra attention when quantizing the model. Most of the recent quantization method packs int2/int4 weights inside `torch.uint8` weights, so this flag should not be really required (set to `False` by default).\n* `is_serializable`: A property method to determine whether the method is serializable or not.\n* `is_trainable`:  A property method to determine whether you can fine-tune models on top of the quantization method (with or without PEFT approaches).\n\n4. Write the `validate_environment` and `update_torch_dtype` methods."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 625437f6-95fd-4c97-9378-2f370c9db7c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/contribute", "title": "Contribute new quantization method", "tokens": 1705, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"625437f6-95fd-4c97-9378-2f370c9db7c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5080c521-e9b9-44e6-951f-5474f9191f04\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53fbf61f9596f0cbe6ceca6e165cb2aa79c2308dae7bcd4afac9e225c76e7e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7dfba02f-86bd-40d8-ba19-c6a7725ec84e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61ce5062dfb13c5b7aa2824d68b6726c30d0d2983420ab5b8b5081c19225dcfb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9ce6b9b9-ded8-4128-81a5-50c7a8d5921a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d9c9e8bcb461c14adc0524f1d6cde9bb70541d415e8e64baf603d27ee423a466\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3193, \"end_char_idx\": 6548, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "doc_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "ref_doc_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "chroma:document": "Make sure to add the new quantizer and quantization config in the quantization auto-mapping in [src/transformers/quantizers/auto.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/auto.py).\n\n3. Define the following class attributes/property methods for your quantization method:\n\n* `requires_calibration`: Whether the quantization method requires a data calibration process. If set to `True`, you can only support inference (with quantized weights) and not inference and quantization.\n* `required_packages`: A list of strings of the required packages to use the quantized weights. You might need to define some new utility methods such as `is_auto_awq_available` in [transformers/src/utils/import_utils.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/utils/import_utils.py).\n* `requires_parameters_quantization`: Only required if your quantization method requires extra attention to the underlying `nn.Parameter` object. For example, bitsandbytes uses `Params4bit` and `Int8Param`, which requires some extra attention when quantizing the model. Most of the recent quantization method packs int2/int4 weights inside `torch.uint8` weights, so this flag should not be really required (set to `False` by default).\n* `is_serializable`: A property method to determine whether the method is serializable or not.\n* `is_trainable`:  A property method to determine whether you can fine-tune models on top of the quantization method (with or without PEFT approaches).\n\n4. Write the `validate_environment` and `update_torch_dtype` methods. These methods are called before creating the quantized model to ensure users use the right configuration. You can have a look at how this is done on other quantizers.\n\n5. Write the `_process_model_before_weight_loading` method. In Transformers, the quantized models are initialized first on the `\"meta\"` device before loading the weights. This means the `_process_model_before_weight_loading` method takes care of manipulating the model skeleton to replace some modules (e.g., `nn.Linear`) with the target modules (quantization modules). You can define a module replacement logic or any other utility method by creating a new file in [transformers/src/integrations/](https://github.com/huggingface/transformers/tree/abbffc4525566a48a9733639797c812301218b83/src/transformers/integrations) and exposing the relevant methods in that folder's `__init__.py` file. The best starting point would be to have a look at another quantization methods such as [quantizer_awq.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/quantizer_awq.py).\n\n6. Write the `_process_model_after_weight_loading` method. This method enables implementing additional features that require manipulating the model after loading the weights.\n\n7. Document everything! Make sure your quantization method is documented by adding a new file under `docs/source/en/quantization` and adding a new row in the table in `docs/source/en/quantization/overview.md`.\n\n8. Add tests! You should add tests by first adding the package in our nightly Dockerfile inside `docker/transformers-quantization-latest-gpu` and then adding a new test file in `tests/quantization/xxx`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9ce6b9b9-ded8-4128-81a5-50c7a8d5921a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/contribute", "title": "Contribute new quantization method", "tokens": 1705, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9ce6b9b9-ded8-4128-81a5-50c7a8d5921a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5080c521-e9b9-44e6-951f-5474f9191f04\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53fbf61f9596f0cbe6ceca6e165cb2aa79c2308dae7bcd4afac9e225c76e7e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"625437f6-95fd-4c97-9378-2f370c9db7c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"efcc377ade78cc2272044bb17ef71350e1a12f77e60fc341596e19e479027083\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4855, \"end_char_idx\": 6625, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "doc_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "ref_doc_id": "5080c521-e9b9-44e6-951f-5474f9191f04", "chroma:document": "These methods are called before creating the quantized model to ensure users use the right configuration. You can have a look at how this is done on other quantizers.\n\n5. Write the `_process_model_before_weight_loading` method. In Transformers, the quantized models are initialized first on the `\"meta\"` device before loading the weights. This means the `_process_model_before_weight_loading` method takes care of manipulating the model skeleton to replace some modules (e.g., `nn.Linear`) with the target modules (quantization modules). You can define a module replacement logic or any other utility method by creating a new file in [transformers/src/integrations/](https://github.com/huggingface/transformers/tree/abbffc4525566a48a9733639797c812301218b83/src/transformers/integrations) and exposing the relevant methods in that folder's `__init__.py` file. The best starting point would be to have a look at another quantization methods such as [quantizer_awq.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/quantizer_awq.py).\n\n6. Write the `_process_model_after_weight_loading` method. This method enables implementing additional features that require manipulating the model after loading the weights.\n\n7. Document everything! Make sure your quantization method is documented by adding a new file under `docs/source/en/quantization` and adding a new row in the table in `docs/source/en/quantization/overview.md`.\n\n8. Add tests! You should add tests by first adding the package in our nightly Dockerfile inside `docker/transformers-quantization-latest-gpu` and then adding a new test file in `tests/quantization/xxx`. Feel free to check out how it is implemented for other quantization methods."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b9ded536-c633-438e-a70e-87c34317f186
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/hqq", "title": "HQQ", "tokens": 738, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b9ded536-c633-438e-a70e-87c34317f186\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/hqq\", \"title\": \"HQQ\", \"tokens\": 738, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f8cdd597-a8dd-4184-bab8-8459aa1aa941\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/hqq\", \"title\": \"HQQ\", \"tokens\": 738, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e7d8fd133e251cc5c8726d994b0d687606ec32f7309c727ecdf1bcb4bf4b1d20\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2161, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f8cdd597-a8dd-4184-bab8-8459aa1aa941", "doc_id": "f8cdd597-a8dd-4184-bab8-8459aa1aa941", "ref_doc_id": "f8cdd597-a8dd-4184-bab8-8459aa1aa941", "chroma:document": "# HQQ \n\nHalf-Quadratic Quantization (HQQ) implements on-the-fly quantization via fast robust optimization. It doesn't require calibration data and can be used to quantize any model.  \nPlease refer to the <a href=\"https://github.com/mobiusml/hqq/\">official package</a> for more details.\n\nFor installation, we recommend you use the following approach to get the latest version and build its corresponding CUDA kernels:\n```\npip install hqq\n```\n\nTo quantize a model, you need to create an [`HqqConfig`]. There are two ways of doing it:\n``` Python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# Method 1: all linear layers will use the same quantization config\nquant_config  = HqqConfig(nbits=8, group_size=64, quant_zero=False, quant_scale=False, axis=0) #axis=0 is used by default\n```\n\n``` Python\n# Method 2: each linear layer with the same tag will use a dedicated quantization config\nq4_config = {'nbits':4, 'group_size':64, 'quant_zero':False, 'quant_scale':False}\nq3_config = {'nbits':3, 'group_size':32, 'quant_zero':False, 'quant_scale':False}\nquant_config  = HqqConfig(dynamic_config={\n  'self_attn.q_proj':q4_config,\n  'self_attn.k_proj':q4_config,\n  'self_attn.v_proj':q4_config,\n  'self_attn.o_proj':q4_config,\n\n  'mlp.gate_proj':q3_config,\n  'mlp.up_proj'  :q3_config,\n  'mlp.down_proj':q3_config,\n})\n```\n\nThe second approach is especially interesting for quantizing Mixture-of-Experts (MoEs) because the experts are less affected by lower quantization settings.\n\n\nThen you simply quantize the model as follows\n``` Python\nmodel = transformers.AutoModelForCausalLM.from_pretrained(\n    model_id, \n    torch_dtype=torch.float16, \n    device_map=\"cuda\", \n    quantization_config=quant_config\n)\n```\n\n## Optimized Runtime\n\nHQQ supports various backends, including pure Pytorch and custom dequantization CUDA kernels. These backends are suitable for older gpus and peft/QLoRA training.\nFor faster inference, HQQ supports 4-bit fused kernels (TorchAO and Marlin), reaching up to 200 tokens/sec on a single 4090.\nFor more details on how to use the backends, please refer to https://github.com/mobiusml/hqq/?tab=readme-ov-file#backend"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1c71f8db-0cb1-4534-97ce-9831257a45fb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/aqlm", "title": "AQLM", "tokens": 838, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1c71f8db-0cb1-4534-97ce-9831257a45fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/aqlm\", \"title\": \"AQLM\", \"tokens\": 838, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"019ab9f1-cfd9-4c85-a2ac-ef028a3d145f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/aqlm\", \"title\": \"AQLM\", \"tokens\": 838, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3cbb09e29701a2808aec57be44154966ecbcded946a32cacee911da3a61bb79\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2512, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "019ab9f1-cfd9-4c85-a2ac-ef028a3d145f", "doc_id": "019ab9f1-cfd9-4c85-a2ac-ef028a3d145f", "ref_doc_id": "019ab9f1-cfd9-4c85-a2ac-ef028a3d145f", "chroma:document": "# AQLM\n\n> [!TIP]\n> Try AQLM on [Google Colab](https://colab.research.google.com/drive/1-xZmBRXT5Fm3Ghn4Mwa2KRypORXb855X?usp=sharing)!\n\nAdditive Quantization of Language Models ([AQLM](https://arxiv.org/abs/2401.06118)) is a Large Language Models compression method. It quantizes multiple weights together and take advantage of interdependencies between them. AQLM represents groups of 8-16 weights as a sum of multiple vector codes.\n\nInference support for AQLM is realised in the `aqlm` library. Make sure to install it to run the models (note aqlm works only with python>=3.10):\n```bash\npip install aqlm[gpu,cpu]\n```\n\nThe library provides efficient kernels for both GPU and CPU inference and training.\n\nThe instructions on how to quantize models yourself, as well as all the relevant code can be found in the corresponding GitHub [repository](https://github.com/Vahe1994/AQLM). To run AQLM models simply load a model that has been quantized with AQLM:\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nquantized_model = AutoModelForCausalLM.from_pretrained(\n    \"ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf\",\n    torch_dtype=\"auto\", \n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf\")\n```\n\n## PEFT\n\nStarting with version `aqlm 1.0.2`, AQLM supports Parameter-Efficient Fine-Tuning in a form of [LoRA](https://huggingface.co/docs/peft/package_reference/lora) integrated into the [PEFT](https://huggingface.co/blog/peft) library.\n\n## AQLM configurations\n\nAQLM quantization setups vary mainly on the number of codebooks used as well as codebook sizes in bits. The most popular setups, as well as inference kernels they support are:\n \n| Kernel | Number of codebooks | Codebook size, bits | Notation | Accuracy | Speedup     | Fast GPU inference | Fast CPU inference |\n|---|---------------------|---------------------|----------|-------------|-------------|--------------------|--------------------|\n| Triton | K                   | N                  | KxN     | -        | Up to ~0.7x | \u2705                  | \u274c                  |\n| CUDA | 1                   | 16                  | 1x16     | Best        | Up to ~1.3x | \u2705                  | \u274c                  |\n| CUDA | 2                   | 8                   | 2x8      | OK          | Up to ~3.0x | \u2705                  | \u274c                  |\n| Numba | K                   | 8                   | Kx8      | Good        | Up to ~4.0x | \u274c                  | \u2705                  |"}


