# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2909c5e8-873a-4d7f-8df8-4ce8ee2214a7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2909c5e8-873a-4d7f-8df8-4ce8ee2214a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bd032317-7053-4db0-a599-81823b8c6938\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b528476e-efda-47a1-a14c-107da9047f76\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3d866c4855c64d690c2f6bcc688f05cb27120e0dd3b023bb984ae415eaf90277\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8b1c2ee6-1a18-41f2-89bf-aec1c55ce330\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"169445644ed42caf1ba5436588d2c8402b6173c0cabb8f8fcd12d55a5845bbbf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6569, \"end_char_idx\": 9291, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bd032317-7053-4db0-a599-81823b8c6938", "doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "ref_doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "chroma:document": "<div class=\"flex gap-4\">\n  <div>\n    <img class=\"rounded-xl\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/quantization/fused_forward_memory_plot.png\" alt=\"generate throughput per batch size\" />\n    <figcaption class=\"mt-2 text-center text-sm text-gray-500\">forward peak memory/batch size</figcaption>\n  </div>\n  <div>\n    <img class=\"rounded-xl\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/quantization/fused_generate_throughput_plot.png\" alt=\"forward latency per batch size\" />\n    <figcaption class=\"mt-2 text-center text-sm text-gray-500\">generate throughput/batch size</figcaption>\n  </div>\n</div>\n\n</hfoption>\n<hfoption id=\"unsupported architectures\">\n\nFor architectures that don't support fused modules yet, you need to create a custom fusing mapping to define which modules need to be fused with the `modules_to_fuse` parameter. For example, to fuse the AWQ modules of the [TheBloke/Yi-34B-AWQ](https://huggingface.co/TheBloke/Yi-34B-AWQ) model.\n\n```python\nimport torch\nfrom transformers import AwqConfig, AutoModelForCausalLM\n\nmodel_id = \"TheBloke/Yi-34B-AWQ\"\n\nquantization_config = AwqConfig(\n    bits=4,\n    fuse_max_seq_len=512,\n    modules_to_fuse={\n        \"attention\": [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n        \"layernorm\": [\"ln1\", \"ln2\", \"norm\"],\n        \"mlp\": [\"gate_proj\", \"up_proj\", \"down_proj\"],\n        \"use_alibi\": False,\n        \"num_attention_heads\": 56,\n        \"num_key_value_heads\": 8,\n        \"hidden_size\": 7168\n    }\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config).to(0)\n```\n\nThe parameter `modules_to_fuse` should include:\n\n- `\"attention\"`: The names of the attention layers to fuse in the following order: query, key, value and output projection layer. If you don't want to fuse these layers, pass an empty list.\n- `\"layernorm\"`: The names of all the LayerNorm layers you want to replace with a custom fused LayerNorm. If you don't want to fuse these layers, pass an empty list.\n- `\"mlp\"`: The names of the MLP layers you want to fuse into a single MLP layer in the order: (gate (dense, layer, post-attention) / up / down layers).\n- `\"use_alibi\"`: If your model uses ALiBi positional embedding.\n- `\"num_attention_heads\"`: The number of attention heads.\n- `\"num_key_value_heads\"`: The number of key value heads that should be used to implement Grouped Query Attention (GQA). If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA), otherwise GQA is used.\n- `\"hidden_size\"`: The dimension of the hidden representations.\n\n</hfoption>\n</hfoptions>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8b1c2ee6-1a18-41f2-89bf-aec1c55ce330
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/awq", "title": "AWQ", "tokens": 3069, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8b1c2ee6-1a18-41f2-89bf-aec1c55ce330\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bd032317-7053-4db0-a599-81823b8c6938\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d91ca270c3c203a3af299772c4d9650c1f6f0148ae8dd6d5378ca1cea8c45eb2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2909c5e8-873a-4d7f-8df8-4ce8ee2214a7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/awq\", \"title\": \"AWQ\", \"tokens\": 3069, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"68109b7c0849fa382ed3db31723ccf697f71a96b9db06056467165b7c42a6a34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8390, \"end_char_idx\": 10456, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bd032317-7053-4db0-a599-81823b8c6938", "doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "ref_doc_id": "bd032317-7053-4db0-a599-81823b8c6938", "chroma:document": "If you don't want to fuse these layers, pass an empty list.\n- `\"layernorm\"`: The names of all the LayerNorm layers you want to replace with a custom fused LayerNorm. If you don't want to fuse these layers, pass an empty list.\n- `\"mlp\"`: The names of the MLP layers you want to fuse into a single MLP layer in the order: (gate (dense, layer, post-attention) / up / down layers).\n- `\"use_alibi\"`: If your model uses ALiBi positional embedding.\n- `\"num_attention_heads\"`: The number of attention heads.\n- `\"num_key_value_heads\"`: The number of key value heads that should be used to implement Grouped Query Attention (GQA). If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA), otherwise GQA is used.\n- `\"hidden_size\"`: The dimension of the hidden representations.\n\n</hfoption>\n</hfoptions>\n\n\n\n## ExLlama-v2 support\n\nRecent versions of `autoawq` supports ExLlama-v2 kernels for faster prefill and decoding. To get started, first install the latest version of `autoawq` by running:\n\n```bash\npip install git+https://github.com/casper-hansen/AutoAWQ.git\n```\n\nGet started by passing an `AwqConfig()` with `version=\"exllama\"`.\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, AwqConfig\n\nquantization_config = AwqConfig(version=\"exllama\")\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke/Mistral-7B-Instruct-v0.1-AWQ\",\n    quantization_config=quantization_config,\n    device_map=\"auto\",\n)\n\ninput_ids = torch.randint(0, 100, (1, 128), dtype=torch.long, device=\"cuda\")\noutput = model(input_ids)\nprint(output.logits)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke/Mistral-7B-Instruct-v0.1-AWQ\")\ninput_ids = tokenizer.encode(\"How to make a cake\", return_tensors=\"pt\").to(model.device)\noutput = model.generate(input_ids, do_sample=True, max_length=50, pad_token_id=50256)\nprint(tokenizer.decode(output[0], skip_special_tokens=True))\n```\n\n<Tip warning={true}>\n\nNote this feature is supported on AMD GPUs.\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 53238ea1-4cb6-437e-a4d6-2141cc3f1eb6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/eetq", "title": "EETQ", "tokens": 573, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"53238ea1-4cb6-437e-a4d6-2141cc3f1eb6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/eetq\", \"title\": \"EETQ\", \"tokens\": 573, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"da2784e2-1fdf-44bf-869d-50339f1028f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/eetq\", \"title\": \"EETQ\", \"tokens\": 573, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d74a9a8dafc086d595ad90c01f199aafb91cc5c44450e6521d11629dceb1926a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1454, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "da2784e2-1fdf-44bf-869d-50339f1028f3", "doc_id": "da2784e2-1fdf-44bf-869d-50339f1028f3", "ref_doc_id": "da2784e2-1fdf-44bf-869d-50339f1028f3", "chroma:document": "# EETQ\n\nThe [EETQ](https://github.com/NetEase-FuXi/EETQ) library supports int8 per-channel weight-only quantization for NVIDIA GPUS. The high-performance GEMM and GEMV kernels are from FasterTransformer and TensorRT-LLM. It requires no calibration dataset and does not need to pre-quantize your model. Moreover, the accuracy degradation is negligible owing to the per-channel quantization. \n\nMake sure you have eetq installed from the [relase page](https://github.com/NetEase-FuXi/EETQ/releases)\n```\npip install --no-cache-dir https://github.com/NetEase-FuXi/EETQ/releases/download/v1.0.0/EETQ-1.0.0+cu121+torch2.1.2-cp310-cp310-linux_x86_64.whl\n```\nor via the source code https://github.com/NetEase-FuXi/EETQ. EETQ requires CUDA capability <= 8.9 and >= 7.0\n```\ngit clone https://github.com/NetEase-FuXi/EETQ.git\ncd EETQ/\ngit submodule update --init --recursive\npip install .\n```\n\nAn unquantized model can be quantized via \"from_pretrained\".\n```py\nfrom transformers import AutoModelForCausalLM, EetqConfig\npath = \"/path/to/model\"\nquantization_config = EetqConfig(\"int8\")\nmodel = AutoModelForCausalLM.from_pretrained(path, device_map=\"auto\", quantization_config=quantization_config)\n```\n\nA quantized model can be saved via \"saved_pretrained\" and be reused again via the \"from_pretrained\".\n\n```py\nquant_path = \"/path/to/save/quantized/model\"\nmodel.save_pretrained(quant_path)\nmodel = AutoModelForCausalLM.from_pretrained(quant_path, device_map=\"auto\")\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0f3f460a-74de-47d8-bd9a-26d467de12dc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0f3f460a-74de-47d8-bd9a-26d467de12dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26e9e2a6-a26c-4763-8146-243c28c80eac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9594b680-e069-4d10-ac64-cd14b57077de\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"266f86726e32079555de7a549f73c9095127dbcaa760024b7fc5233d5b49b803\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2518, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "ref_doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "chroma:document": "# bitsandbytes\n\n[bitsandbytes](https://github.com/TimDettmers/bitsandbytes) is the easiest option for quantizing a model to 8 and 4-bit. 8-bit quantization multiplies outliers in fp16 with non-outliers in int8, converts the non-outlier values back to fp16, and then adds them together to return the weights in fp16. This reduces the degradative effect outlier values have on a model's performance. 4-bit quantization compresses a model even further, and it is commonly used with [QLoRA](https://hf.co/papers/2305.14314) to finetune quantized LLMs.\n\nTo use bitsandbytes, make sure you have the following libraries installed:\n\n<hfoptions id=\"bnb\">\n<hfoption id=\"8-bit\">\n\n```bash\npip install transformers accelerate bitsandbytes>0.37.0\n```\n\n</hfoption>\n<hfoption id=\"4-bit\">\n\n```bash\npip install bitsandbytes>=0.39.0\npip install --upgrade accelerate transformers\n```\n\n</hfoption>\n</hfoptions>\n\nNow you can quantize a model by passing a `BitsAndBytesConfig` to [`~PreTrainedModel.from_pretrained`] method. This works for any model in any modality, as long as it supports loading with Accelerate and contains `torch.nn.Linear` layers.\n\n<hfoptions id=\"bnb\">\n<hfoption id=\"8-bit\">\n\nQuantizing a model in 8-bit halves the memory-usage, and for large models, set `device_map=\"auto\"` to efficiently use the GPUs available:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\", \n    quantization_config=quantization_config\n)\n```\n\nBy default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"facebook/opt-350m\", \n    quantization_config=quantization_config, \n    torch_dtype=torch.float32\n)\nmodel_8bit.model.decoder.layers[-1].final_layer_norm.weight.dtype\n```\n\nOnce a model is quantized to 8-bit, you can't push the quantized weights to the Hub unless you're using the latest version of Transformers and bitsandbytes. If you have the latest versions, then you can push the 8-bit model to the Hub with the [`~PreTrainedModel.push_to_hub`] method. The quantization config.json file is pushed first, followed by the quantized model weights."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9594b680-e069-4d10-ac64-cd14b57077de
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9594b680-e069-4d10-ac64-cd14b57077de\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26e9e2a6-a26c-4763-8146-243c28c80eac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0f3f460a-74de-47d8-bd9a-26d467de12dc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b551169779fb85136adf9528a69abfc464db0a434f72794d725b2a51d04f9258\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f4f59299-4ecc-46bf-a97b-e6e2402f620d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a3fd63bcb35a84fd67fd2d9f9552ef3eabb408a5abc8b7373dc6ceaf9388b653\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1002, \"end_char_idx\": 4010, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "ref_doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "chroma:document": "This works for any model in any modality, as long as it supports loading with Accelerate and contains `torch.nn.Linear` layers.\n\n<hfoptions id=\"bnb\">\n<hfoption id=\"8-bit\">\n\nQuantizing a model in 8-bit halves the memory-usage, and for large models, set `device_map=\"auto\"` to efficiently use the GPUs available:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\", \n    quantization_config=quantization_config\n)\n```\n\nBy default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"facebook/opt-350m\", \n    quantization_config=quantization_config, \n    torch_dtype=torch.float32\n)\nmodel_8bit.model.decoder.layers[-1].final_layer_norm.weight.dtype\n```\n\nOnce a model is quantized to 8-bit, you can't push the quantized weights to the Hub unless you're using the latest version of Transformers and bitsandbytes. If you have the latest versions, then you can push the 8-bit model to the Hub with the [`~PreTrainedModel.push_to_hub`] method. The quantization config.json file is pushed first, followed by the quantized model weights.\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-560m\", \n    quantization_config=quantization_config\n)\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/bloom-560m\")\n\nmodel.push_to_hub(\"bloom-560m-8bit\")\n```\n\n</hfoption>\n<hfoption id=\"4-bit\">\n\nQuantizing a model in 4-bit reduces your memory-usage by 4x, and for large models, set `device_map=\"auto\"` to efficiently use the GPUs available:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_4bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\",\n    quantization_config=quantization_config\n)\n```\n\nBy default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_4bit = AutoModelForCausalLM.from_pretrained(\n    \"facebook/opt-350m\",\n    quantization_config=quantization_config, \n    torch_dtype=torch.float32\n)\nmodel_4bit.model.decoder.layers[-1].final_layer_norm.weight.dtype\n```\n\nIf you have `bitsandbytes>=0.41.3`, you can serialize 4-bit models and push them on Hugging Face Hub."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f4f59299-4ecc-46bf-a97b-e6e2402f620d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f4f59299-4ecc-46bf-a97b-e6e2402f620d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26e9e2a6-a26c-4763-8146-243c28c80eac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9594b680-e069-4d10-ac64-cd14b57077de\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"81baaec93aa50717899447d49ceeb8117fbc53fd75d07526464d47e0825205c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"16ee9cf6-1d28-4931-9e39-46c36bb76acf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"210688437e42442c50efbb95e26ade4244b2a2b0ab508a713687be81cdcd7660\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2520, \"end_char_idx\": 5338, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "ref_doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "chroma:document": "```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-560m\", \n    quantization_config=quantization_config\n)\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/bloom-560m\")\n\nmodel.push_to_hub(\"bloom-560m-8bit\")\n```\n\n</hfoption>\n<hfoption id=\"4-bit\">\n\nQuantizing a model in 4-bit reduces your memory-usage by 4x, and for large models, set `device_map=\"auto\"` to efficiently use the GPUs available:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_4bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\",\n    quantization_config=quantization_config\n)\n```\n\nBy default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_4bit = AutoModelForCausalLM.from_pretrained(\n    \"facebook/opt-350m\",\n    quantization_config=quantization_config, \n    torch_dtype=torch.float32\n)\nmodel_4bit.model.decoder.layers[-1].final_layer_norm.weight.dtype\n```\n\nIf you have `bitsandbytes>=0.41.3`, you can serialize 4-bit models and push them on Hugging Face Hub. Simply call `model.push_to_hub()` after loading it in 4-bit precision. You can also save the serialized 4-bit models locally with `model.save_pretrained()` command.  \n\n</hfoption>\n</hfoptions>\n\n<Tip warning={true}>\n\nTraining with 8-bit and 4-bit weights are only supported for training *extra* parameters.\n\n</Tip>\n\nYou can check your memory footprint with the `get_memory_footprint` method:\n\n```py\nprint(model.get_memory_footprint())\n```\n\nQuantized models can be loaded from the [`~PreTrainedModel.from_pretrained`] method without needing to specify the `load_in_8bit` or `load_in_4bit` parameters:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/bloom-560m-8bit\", device_map=\"auto\")\n```\n\n## 8-bit (LLM.int8() algorithm)\n\n<Tip>\n\nLearn more about the details of 8-bit quantization in this [blog post](https://huggingface.co/blog/hf-bitsandbytes-integration)!\n\n</Tip>\n\nThis section explores some of the specific features of 8-bit models, such as offloading, outlier thresholds, skipping module conversion, and finetuning.\n\n### Offloading\n\n8-bit models can offload weights between the CPU and GPU to support fitting very large models into memory. The weights dispatched to the CPU are actually stored in **float32**, and aren't converted to 8-bit."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 16ee9cf6-1d28-4931-9e39-46c36bb76acf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"16ee9cf6-1d28-4931-9e39-46c36bb76acf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26e9e2a6-a26c-4763-8146-243c28c80eac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f4f59299-4ecc-46bf-a97b-e6e2402f620d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ccffdaa968a8352e3b3b27ff4df3c218bd014c226acfdb11ce1ba8676d7af3a7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cb74aa10-1398-40f8-8501-f23d4dd32916\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6a927d16b20be179139b6a848267bcad9848d3f412928814c567cd30b5287467\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4011, \"end_char_idx\": 6716, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "ref_doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "chroma:document": "Simply call `model.push_to_hub()` after loading it in 4-bit precision. You can also save the serialized 4-bit models locally with `model.save_pretrained()` command.  \n\n</hfoption>\n</hfoptions>\n\n<Tip warning={true}>\n\nTraining with 8-bit and 4-bit weights are only supported for training *extra* parameters.\n\n</Tip>\n\nYou can check your memory footprint with the `get_memory_footprint` method:\n\n```py\nprint(model.get_memory_footprint())\n```\n\nQuantized models can be loaded from the [`~PreTrainedModel.from_pretrained`] method without needing to specify the `load_in_8bit` or `load_in_4bit` parameters:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/bloom-560m-8bit\", device_map=\"auto\")\n```\n\n## 8-bit (LLM.int8() algorithm)\n\n<Tip>\n\nLearn more about the details of 8-bit quantization in this [blog post](https://huggingface.co/blog/hf-bitsandbytes-integration)!\n\n</Tip>\n\nThis section explores some of the specific features of 8-bit models, such as offloading, outlier thresholds, skipping module conversion, and finetuning.\n\n### Offloading\n\n8-bit models can offload weights between the CPU and GPU to support fitting very large models into memory. The weights dispatched to the CPU are actually stored in **float32**, and aren't converted to 8-bit. For example, to enable offloading for the [bigscience/bloom-1b7](https://huggingface.co/bigscience/bloom-1b7) model, start by creating a [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(llm_int8_enable_fp32_cpu_offload=True)\n```\n\nDesign a custom device map to fit everything on your GPU except for the `lm_head`, which you'll dispatch to the CPU:\n\n```py\ndevice_map = {\n    \"transformer.word_embeddings\": 0,\n    \"transformer.word_embeddings_layernorm\": 0,\n    \"lm_head\": \"cpu\",\n    \"transformer.h\": 0,\n    \"transformer.ln_f\": 0,\n}\n```\n\nNow load your model with the custom `device_map` and `quantization_config`:\n\n```py\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\",\n    device_map=device_map,\n    quantization_config=quantization_config,\n)\n```\n\n### Outlier threshold\n\nAn \"outlier\" is a hidden state value greater than a certain threshold, and these values are computed in fp16. While the values are usually normally distributed ([-3.5, 3.5]), this distribution can be very different for large models ([-60, 6] or [6, 60]). 8-bit quantization works well for values ~5, but beyond that, there is a significant performance penalty. A good default threshold value is 6, but a lower threshold may be needed for more unstable models (small models or finetuning)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cb74aa10-1398-40f8-8501-f23d4dd32916
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cb74aa10-1398-40f8-8501-f23d4dd32916\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26e9e2a6-a26c-4763-8146-243c28c80eac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"16ee9cf6-1d28-4931-9e39-46c36bb76acf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"86f9284f698a85144003641758f55b67df56b1d7ff7bde20043ff037fa304f50\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b7390a0b-540a-4867-a05c-130557406250\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cba71e72f8eff09b4489101ac527b10c6629832bec214ca7af56b88c69a5b184\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5238, \"end_char_idx\": 8103, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "ref_doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "chroma:document": "The weights dispatched to the CPU are actually stored in **float32**, and aren't converted to 8-bit. For example, to enable offloading for the [bigscience/bloom-1b7](https://huggingface.co/bigscience/bloom-1b7) model, start by creating a [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(llm_int8_enable_fp32_cpu_offload=True)\n```\n\nDesign a custom device map to fit everything on your GPU except for the `lm_head`, which you'll dispatch to the CPU:\n\n```py\ndevice_map = {\n    \"transformer.word_embeddings\": 0,\n    \"transformer.word_embeddings_layernorm\": 0,\n    \"lm_head\": \"cpu\",\n    \"transformer.h\": 0,\n    \"transformer.ln_f\": 0,\n}\n```\n\nNow load your model with the custom `device_map` and `quantization_config`:\n\n```py\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    \"bigscience/bloom-1b7\",\n    device_map=device_map,\n    quantization_config=quantization_config,\n)\n```\n\n### Outlier threshold\n\nAn \"outlier\" is a hidden state value greater than a certain threshold, and these values are computed in fp16. While the values are usually normally distributed ([-3.5, 3.5]), this distribution can be very different for large models ([-60, 6] or [6, 60]). 8-bit quantization works well for values ~5, but beyond that, there is a significant performance penalty. A good default threshold value is 6, but a lower threshold may be needed for more unstable models (small models or finetuning).\n\nTo find the best threshold for your model, we recommend experimenting with the `llm_int8_threshold` parameter in [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nmodel_id = \"bigscience/bloom-1b7\"\n\nquantization_config = BitsAndBytesConfig(\n    llm_int8_threshold=10,\n)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=device_map,\n    quantization_config=quantization_config,\n)\n```\n\n### Skip module conversion\n\nFor some models, like [Jukebox](model_doc/jukebox), you don't need to quantize every module to 8-bit which can actually cause instability. With Jukebox, there are several `lm_head` modules that should be skipped using the `llm_int8_skip_modules` parameter in [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nmodel_id = \"bigscience/bloom-1b7\"\n\nquantization_config = BitsAndBytesConfig(\n    llm_int8_skip_modules=[\"lm_head\"],\n)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=\"auto\",\n    quantization_config=quantization_config,\n)\n```\n\n### Finetuning\n\nWith the [PEFT](https://github.com/huggingface/peft) library, you can finetune large models like [flan-t5-large](https://huggingface.co/google/flan-t5-large) and [facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b) with 8-bit quantization."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b7390a0b-540a-4867-a05c-130557406250
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b7390a0b-540a-4867-a05c-130557406250\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26e9e2a6-a26c-4763-8146-243c28c80eac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb74aa10-1398-40f8-8501-f23d4dd32916\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"64f94c8389347505f7252aa7a848f0ceafdf7977291077e5b90f33d44cb9003c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"37b463d5-616d-4122-8b2e-046a44865369\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a536b22889ac87da0c70b6f9b9995d33bf2d62f286a95bd0c40d6878716fc9e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6589, \"end_char_idx\": 8831, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "ref_doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "chroma:document": "A good default threshold value is 6, but a lower threshold may be needed for more unstable models (small models or finetuning).\n\nTo find the best threshold for your model, we recommend experimenting with the `llm_int8_threshold` parameter in [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nmodel_id = \"bigscience/bloom-1b7\"\n\nquantization_config = BitsAndBytesConfig(\n    llm_int8_threshold=10,\n)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=device_map,\n    quantization_config=quantization_config,\n)\n```\n\n### Skip module conversion\n\nFor some models, like [Jukebox](model_doc/jukebox), you don't need to quantize every module to 8-bit which can actually cause instability. With Jukebox, there are several `lm_head` modules that should be skipped using the `llm_int8_skip_modules` parameter in [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nmodel_id = \"bigscience/bloom-1b7\"\n\nquantization_config = BitsAndBytesConfig(\n    llm_int8_skip_modules=[\"lm_head\"],\n)\n\nmodel_8bit = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=\"auto\",\n    quantization_config=quantization_config,\n)\n```\n\n### Finetuning\n\nWith the [PEFT](https://github.com/huggingface/peft) library, you can finetune large models like [flan-t5-large](https://huggingface.co/google/flan-t5-large) and [facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b) with 8-bit quantization. You don't need to pass the `device_map` parameter for training because it'll automatically load your model on a GPU. However, you can still customize the device map with the `device_map` parameter if you want to (`device_map=\"auto\"` should only be used for inference).\n\n## 4-bit (QLoRA algorithm)\n\n<Tip>\n\nTry 4-bit quantization in this [notebook](https://colab.research.google.com/drive/1ge2F1QSK8Q7h0hn3YKuBCOAS0bK8E0wf) and learn more about it's details in this [blog post](https://huggingface.co/blog/4bit-transformers-bitsandbytes).\n\n</Tip>\n\nThis section explores some of the specific features of 4-bit models, such as changing the compute data type, using the Normal Float 4 (NF4) data type, and using nested quantization."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 37b463d5-616d-4122-8b2e-046a44865369
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/bitsandbytes", "title": "bitsandbytes", "tokens": 3097, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"37b463d5-616d-4122-8b2e-046a44865369\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26e9e2a6-a26c-4763-8146-243c28c80eac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e79a582bb98a15c206cf9786fb60f4f93e72d57134978dc23c235a6e0e9271\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7390a0b-540a-4867-a05c-130557406250\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/bitsandbytes\", \"title\": \"bitsandbytes\", \"tokens\": 3097, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"779d5c356a38d788a7894c7d16025dd26139b806da797b84f8ef97c51d69fa34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8104, \"end_char_idx\": 11387, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "ref_doc_id": "26e9e2a6-a26c-4763-8146-243c28c80eac", "chroma:document": "You don't need to pass the `device_map` parameter for training because it'll automatically load your model on a GPU. However, you can still customize the device map with the `device_map` parameter if you want to (`device_map=\"auto\"` should only be used for inference).\n\n## 4-bit (QLoRA algorithm)\n\n<Tip>\n\nTry 4-bit quantization in this [notebook](https://colab.research.google.com/drive/1ge2F1QSK8Q7h0hn3YKuBCOAS0bK8E0wf) and learn more about it's details in this [blog post](https://huggingface.co/blog/4bit-transformers-bitsandbytes).\n\n</Tip>\n\nThis section explores some of the specific features of 4-bit models, such as changing the compute data type, using the Normal Float 4 (NF4) data type, and using nested quantization.\n\n\n### Compute data type\n\nTo speedup computation, you can change the data type from float32 (the default value) to bf16 using the `bnb_4bit_compute_dtype` parameter in [`BitsAndBytesConfig`]:\n\n```py\nimport torch\nfrom transformers import BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)\n```\n\n### Normal Float 4 (NF4)\n\nNF4 is a 4-bit data type from the [QLoRA](https://hf.co/papers/2305.14314) paper, adapted for weights initialized from a normal distribution. You should use NF4 for training 4-bit base models. This can be configured with the `bnb_4bit_quant_type` parameter in the [`BitsAndBytesConfig`]:\n\n```py\nfrom transformers import BitsAndBytesConfig\n\nnf4_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n)\n\nmodel_nf4 = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=nf4_config)\n```\n\nFor inference, the `bnb_4bit_quant_type` does not have a huge impact on performance. However, to remain consistent with the model weights, you should use the `bnb_4bit_compute_dtype` and `torch_dtype` values.\n\n### Nested quantization\n\nNested quantization is a technique that can save additional memory at no additional performance cost. This feature performs a second quantization of the already quantized weights to save an addition 0.4 bits/parameter. For example, with nested quantization, you can finetune a [Llama-13b](https://huggingface.co/meta-llama/Llama-2-13b) model on a 16GB NVIDIA T4 GPU with a sequence length of 1024, a batch size of 1, and enabling gradient accumulation with 4 steps.\n\n```py\nfrom transformers import BitsAndBytesConfig\n\ndouble_quant_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_use_double_quant=True,\n)\n\nmodel_double_quant = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-2-13b\", quantization_config=double_quant_config)\n```\n\n## Dequantizing `bitsandbytes` models\n\nOnce quantized, you can dequantize the model to the original precision but this might result in a small quality loss of the model. Make sure you have enough GPU RAM to fit the dequantized model. \n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer\n\nmodel_id = \"facebook/opt-125m\"\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, BitsAndBytesConfig(load_in_4bit=True))\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\nmodel.dequantize()\n\ntext = tokenizer(\"Hello my name is\", return_tensors=\"pt\").to(0)\n\nout = model.generate(**text)\nprint(tokenizer.decode(out[0]))\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 354059b3-ce7b-45ab-aa60-303ef8c55710
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/torchao", "title": "TorchAO", "tokens": 609, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"354059b3-ce7b-45ab-aa60-303ef8c55710\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/torchao\", \"title\": \"TorchAO\", \"tokens\": 609, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fde100d1-206c-4c9e-b125-e41f52e7b40f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/torchao\", \"title\": \"TorchAO\", \"tokens\": 609, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1760ec8398570846a930d2677dc1779b6387c098264db3793d710e0ae5704051\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1930, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fde100d1-206c-4c9e-b125-e41f52e7b40f", "doc_id": "fde100d1-206c-4c9e-b125-e41f52e7b40f", "ref_doc_id": "fde100d1-206c-4c9e-b125-e41f52e7b40f", "chroma:document": "# TorchAO\n\n[TorchAO](https://github.com/pytorch/ao) is an architecture optimization library for PyTorch, it provides high performance dtypes, optimization techniques and kernels for inference and training, featuring composability with native PyTorch features like `torch.compile`, FSDP etc.. Some benchmark numbers can be found [here](https://github.com/pytorch/ao/tree/main?tab=readme-ov-file#without-intrusive-code-changes)\n\nBefore you begin, make sure the following libraries are installed with their latest version:\n\n```bash\npip install --upgrade torch torchao\n```\n\n\n```py\nfrom transformers import TorchAoConfig, AutoModelForCausalLM, AutoTokenizer\n\nmodel_name = \"meta-llama/Meta-Llama-3-8B\"\n# We support int4_weight_only, int8_weight_only and int8_dynamic_activation_int8_weight\n# More examples and documentations for arguments can be found in https://github.com/pytorch/ao/tree/main/torchao/quantization#other-available-quantization-techniques\nquantization_config = TorchAoConfig(\"int4_weight_only\", group_size=128)\nquantized_model = AutoModelForCausalLM.from_pretrained(model_name, device_map=\"auto\", quantization_config=quantization_config)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\ninput_text = \"What are we having for dinner?\"\ninput_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cuda\")\n\n# compile the quantizd model to get speedup\nimport torchao\ntorchao.quantization.utils.recommended_inductor_config_setter()\nquantized_model = torch.compile(quantized_model, mode=\"max-autotune\")\n\noutput = quantized_model.generate(**input_ids, max_new_tokens=10)\nprint(tokenizer.decode(output[0], skip_special_tokens=True))\n```\n\ntorchao quantization is implemented with tensor subclasses, currently it does not work with huggingface serialization, both the safetensor option and [non-safetensor option](https://github.com/huggingface/transformers/issues/32364), we'll update here with instructions when it's working."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 70b3d88f-11b9-45f6-b401-9bb2f9e2d5c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/gptq", "title": "GPTQ", "tokens": 1683, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"70b3d88f-11b9-45f6-b401-9bb2f9e2d5c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5329ace6-9220-4928-86a7-9a5ad87631d2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a479a124df2133faab587f566c0a3c83ef18561238ec1c24a6952fc12a81e8d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2d31a32d-902d-41c9-a9fc-e3f75a3300ce\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0cc1d88ad2a9c0a58df133850f5386334af1047093d0971ebeac45f1bad97f6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2978, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "doc_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "ref_doc_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "chroma:document": "# GPTQ\n\n<Tip>\n\nTry GPTQ quantization with PEFT in this [notebook](https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb?usp=sharing) and learn more about it's details in this [blog post](https://huggingface.co/blog/gptq-integration)!\n\n</Tip>\n\nThe [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) library implements the GPTQ algorithm, a post-training quantization technique where each row of the weight matrix is quantized independently to find a version of the weights that minimizes the error. These weights are quantized to int4, but they're restored to fp16 on the fly during inference. This can save your memory-usage by 4x because the int4 weights are dequantized in a fused kernel rather than a GPU's global memory, and you can also expect a speedup in inference because using a lower bitwidth takes less time to communicate.\n\nBefore you begin, make sure the following libraries are installed:\n\n```bash\npip install auto-gptq\npip install --upgrade accelerate optimum transformers\n```\n\nTo quantize a model (currently only supported for text models), you need to create a [`GPTQConfig`] class and set the number of bits to quantize to, a dataset to calibrate the weights for quantization, and a tokenizer to prepare the dataset.\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig\n\nmodel_id = \"facebook/opt-125m\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\ngptq_config = GPTQConfig(bits=4, dataset=\"c4\", tokenizer=tokenizer)\n```\n\nYou could also pass your own dataset as a list of strings, but it is highly recommended to use the same dataset from the GPTQ paper.\n\n```py\ndataset = [\"auto-gptq is an easy-to-use model quantization library with user-friendly apis, based on GPTQ algorithm.\"]\ngptq_config = GPTQConfig(bits=4, dataset=dataset, tokenizer=tokenizer)\n```\n\nLoad a model to quantize and pass the `gptq_config` to the [`~AutoModelForCausalLM.from_pretrained`] method. Set `device_map=\"auto\"` to automatically offload the model to a CPU to help fit the model in memory, and allow the model modules to be moved between the CPU and GPU for quantization.\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"auto\", quantization_config=gptq_config)\n```\n\nIf you're running out of memory because a dataset is too large, disk offloading is not supported. If this is the case, try passing the `max_memory` parameter to allocate the amount of memory to use on your device (GPU and CPU):\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"auto\", max_memory={0: \"30GiB\", 1: \"46GiB\", \"cpu\": \"30GiB\"}, quantization_config=gptq_config)\n```\n\n<Tip warning={true}>\n\nDepending on your hardware, it can take some time to quantize a model from scratch. It can take ~5 minutes to quantize the [facebook/opt-350m](https://huggingface.co/facebook/opt-350m) model on a free-tier Google Colab GPU, but it'll take ~4 hours to quantize a 175B parameter model on a NVIDIA A100."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2d31a32d-902d-41c9-a9fc-e3f75a3300ce
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/gptq", "title": "GPTQ", "tokens": 1683, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2d31a32d-902d-41c9-a9fc-e3f75a3300ce\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5329ace6-9220-4928-86a7-9a5ad87631d2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a479a124df2133faab587f566c0a3c83ef18561238ec1c24a6952fc12a81e8d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"70b3d88f-11b9-45f6-b401-9bb2f9e2d5c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"459e8c30c5a6b0b92fb69565dc838e8a07e0ba1d76ad5d75dd345c7aef72f50c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"90bf5987-48ca-4ad4-b2ff-cb96f9df1c6b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fbc7ac6e9009608e4090199c0c028b6df1f84552c7d60767fd91a39e8391cc01\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1624, \"end_char_idx\": 4121, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "doc_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "ref_doc_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "chroma:document": "```py\ndataset = [\"auto-gptq is an easy-to-use model quantization library with user-friendly apis, based on GPTQ algorithm.\"]\ngptq_config = GPTQConfig(bits=4, dataset=dataset, tokenizer=tokenizer)\n```\n\nLoad a model to quantize and pass the `gptq_config` to the [`~AutoModelForCausalLM.from_pretrained`] method. Set `device_map=\"auto\"` to automatically offload the model to a CPU to help fit the model in memory, and allow the model modules to be moved between the CPU and GPU for quantization.\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"auto\", quantization_config=gptq_config)\n```\n\nIf you're running out of memory because a dataset is too large, disk offloading is not supported. If this is the case, try passing the `max_memory` parameter to allocate the amount of memory to use on your device (GPU and CPU):\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"auto\", max_memory={0: \"30GiB\", 1: \"46GiB\", \"cpu\": \"30GiB\"}, quantization_config=gptq_config)\n```\n\n<Tip warning={true}>\n\nDepending on your hardware, it can take some time to quantize a model from scratch. It can take ~5 minutes to quantize the [facebook/opt-350m](https://huggingface.co/facebook/opt-350m) model on a free-tier Google Colab GPU, but it'll take ~4 hours to quantize a 175B parameter model on a NVIDIA A100. Before you quantize a model, it is a good idea to check the Hub if a GPTQ-quantized version of the model already exists.\n\n</Tip>\n\nOnce your model is quantized, you can push the model and tokenizer to the Hub where it can be easily shared and accessed. Use the [`~PreTrainedModel.push_to_hub`] method to save the [`GPTQConfig`]:\n\n```py\nquantized_model.push_to_hub(\"opt-125m-gptq\")\ntokenizer.push_to_hub(\"opt-125m-gptq\")\n```\n\nYou could also save your quantized model locally with the [`~PreTrainedModel.save_pretrained`] method. If the model was quantized with the `device_map` parameter, make sure to move the entire model to a GPU or CPU before saving it. For example, to save the model on a CPU:\n\n```py\nquantized_model.save_pretrained(\"opt-125m-gptq\")\ntokenizer.save_pretrained(\"opt-125m-gptq\")\n\n# if quantized with device_map set\nquantized_model.to(\"cpu\")\nquantized_model.save_pretrained(\"opt-125m-gptq\")\n```\n\nReload a quantized model with the [`~PreTrainedModel.from_pretrained`] method, and set `device_map=\"auto\"` to automatically distribute the model on all available GPUs to load the model faster without using more memory than needed."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 90bf5987-48ca-4ad4-b2ff-cb96f9df1c6b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/gptq", "title": "GPTQ", "tokens": 1683, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"90bf5987-48ca-4ad4-b2ff-cb96f9df1c6b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5329ace6-9220-4928-86a7-9a5ad87631d2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a479a124df2133faab587f566c0a3c83ef18561238ec1c24a6952fc12a81e8d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2d31a32d-902d-41c9-a9fc-e3f75a3300ce\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8c7108e1a4d5c3cbee68a082ac4c461296b20ff4f9471a4f658df08b62674deb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c26ec99d-9d7f-451a-a8cc-6761ef945be8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"75cd28e5f9db26468624fcd5dff81c8e48fa5ef452717abc0192fd5b68a74ca5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2762, \"end_char_idx\": 5571, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "doc_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "ref_doc_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "chroma:document": "It can take ~5 minutes to quantize the [facebook/opt-350m](https://huggingface.co/facebook/opt-350m) model on a free-tier Google Colab GPU, but it'll take ~4 hours to quantize a 175B parameter model on a NVIDIA A100. Before you quantize a model, it is a good idea to check the Hub if a GPTQ-quantized version of the model already exists.\n\n</Tip>\n\nOnce your model is quantized, you can push the model and tokenizer to the Hub where it can be easily shared and accessed. Use the [`~PreTrainedModel.push_to_hub`] method to save the [`GPTQConfig`]:\n\n```py\nquantized_model.push_to_hub(\"opt-125m-gptq\")\ntokenizer.push_to_hub(\"opt-125m-gptq\")\n```\n\nYou could also save your quantized model locally with the [`~PreTrainedModel.save_pretrained`] method. If the model was quantized with the `device_map` parameter, make sure to move the entire model to a GPU or CPU before saving it. For example, to save the model on a CPU:\n\n```py\nquantized_model.save_pretrained(\"opt-125m-gptq\")\ntokenizer.save_pretrained(\"opt-125m-gptq\")\n\n# if quantized with device_map set\nquantized_model.to(\"cpu\")\nquantized_model.save_pretrained(\"opt-125m-gptq\")\n```\n\nReload a quantized model with the [`~PreTrainedModel.from_pretrained`] method, and set `device_map=\"auto\"` to automatically distribute the model on all available GPUs to load the model faster without using more memory than needed.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"auto\")\n```\n\n## ExLlama\n\n[ExLlama](https://github.com/turboderp/exllama) is a Python/C++/CUDA implementation of the [Llama](model_doc/llama) model that is designed for faster inference with 4-bit GPTQ weights (check out these [benchmarks](https://github.com/huggingface/optimum/tree/main/tests/benchmark#gptq-benchmark)). The ExLlama kernel is activated by default when you create a [`GPTQConfig`] object. To boost inference speed even further, use the [ExLlamaV2](https://github.com/turboderp/exllamav2) kernels by configuring the `exllama_config` parameter:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, GPTQConfig\n\ngptq_config = GPTQConfig(bits=4, exllama_config={\"version\":2})\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"auto\", quantization_config=gptq_config)\n```\n\n<Tip warning={true}>\n\nOnly 4-bit models are supported, and we recommend deactivating the ExLlama kernels if you're finetuning a quantized model with PEFT.\n\n</Tip>\n\nThe ExLlama kernels are only supported when the entire model is on the GPU. If you're doing inference on a CPU with AutoGPTQ (version > 0.4.2), then you'll need to disable the ExLlama kernel. This overwrites the attributes related to the ExLlama kernels in the quantization config of the config.json file."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c26ec99d-9d7f-451a-a8cc-6761ef945be8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/gptq", "title": "GPTQ", "tokens": 1683, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c26ec99d-9d7f-451a-a8cc-6761ef945be8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5329ace6-9220-4928-86a7-9a5ad87631d2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a479a124df2133faab587f566c0a3c83ef18561238ec1c24a6952fc12a81e8d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"90bf5987-48ca-4ad4-b2ff-cb96f9df1c6b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/gptq\", \"title\": \"GPTQ\", \"tokens\": 1683, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"49e50f6879e1c3910a618ba4a25a79ff1bce16ceefb13ba3b758c56db8c5fd57\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4123, \"end_char_idx\": 5834, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "doc_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "ref_doc_id": "5329ace6-9220-4928-86a7-9a5ad87631d2", "chroma:document": "```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"auto\")\n```\n\n## ExLlama\n\n[ExLlama](https://github.com/turboderp/exllama) is a Python/C++/CUDA implementation of the [Llama](model_doc/llama) model that is designed for faster inference with 4-bit GPTQ weights (check out these [benchmarks](https://github.com/huggingface/optimum/tree/main/tests/benchmark#gptq-benchmark)). The ExLlama kernel is activated by default when you create a [`GPTQConfig`] object. To boost inference speed even further, use the [ExLlamaV2](https://github.com/turboderp/exllamav2) kernels by configuring the `exllama_config` parameter:\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, GPTQConfig\n\ngptq_config = GPTQConfig(bits=4, exllama_config={\"version\":2})\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"auto\", quantization_config=gptq_config)\n```\n\n<Tip warning={true}>\n\nOnly 4-bit models are supported, and we recommend deactivating the ExLlama kernels if you're finetuning a quantized model with PEFT.\n\n</Tip>\n\nThe ExLlama kernels are only supported when the entire model is on the GPU. If you're doing inference on a CPU with AutoGPTQ (version > 0.4.2), then you'll need to disable the ExLlama kernel. This overwrites the attributes related to the ExLlama kernels in the quantization config of the config.json file.\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM, GPTQConfig\ngptq_config = GPTQConfig(bits=4, use_exllama=False)\nmodel = AutoModelForCausalLM.from_pretrained(\"{your_username}/opt-125m-gptq\", device_map=\"cpu\", quantization_config=gptq_config)\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2847c2a4-2b4a-4e0d-8b5a-f237262ae5b0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/overview", "title": "Quantization", "tokens": 1420, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2847c2a4-2b4a-4e0d-8b5a-f237262ae5b0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1420, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"db5af843-fb90-4c5d-8fdc-280c72d4586e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1420, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e75db12f43c89e8b8c8cf03490026c59210b8d9a659ff32dba2f66b319f926d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4944f3a9-2e78-431f-bd2a-41a5b5bccde5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2735bf9c52eee36adac64226ccf3bd65c6323aa8b11b93eb88a57abb74a7a9c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3515, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "db5af843-fb90-4c5d-8fdc-280c72d4586e", "doc_id": "db5af843-fb90-4c5d-8fdc-280c72d4586e", "ref_doc_id": "db5af843-fb90-4c5d-8fdc-280c72d4586e", "chroma:document": "# Quantization\n\nQuantization techniques focus on representing data with less information while also trying to not lose too much accuracy. This often means converting a data type to represent the same information with fewer bits. For example, if your model weights are stored as 32-bit floating points and they're quantized to 16-bit floating points, this halves the model size which makes it easier to store and reduces memory-usage. Lower precision can also speedup inference because it takes less time to perform calculations with fewer bits.\n\n<Tip>\n\nInterested in adding a new quantization method to Transformers? Read the [HfQuantizer](./contribute) guide to learn how!\n\n</Tip>\n\n<Tip>\n\nIf you are new to the quantization field, we recommend you to check out these beginner-friendly courses about quantization in collaboration with DeepLearning.AI:\n\n* [Quantization Fundamentals with Hugging Face](https://www.deeplearning.ai/short-courses/quantization-fundamentals-with-hugging-face/)\n* [Quantization in Depth](https://www.deeplearning.ai/short-courses/quantization-in-depth/)\n\n</Tip>\n\n## When to use what?\n\nThe community has developed many quantization methods for various use cases. With Transformers, you can run any of these integrated methods depending on your use case because each method has their own pros and cons.\n\nFor example, some quantization methods require calibrating the model with a dataset for more accurate and \"extreme\" compression (up to 1-2 bits quantization), while other methods work out of the box with on-the-fly quantization.\n\nAnother parameter to consider is compatibility with your target device. Do you want to quantize on a CPU, GPU, or Apple silicon?\n\nIn short, supporting a wide range of quantization methods allows you to pick the best quantization method for your specific use case.\n\nUse the table below to help you decide which quantization method to use.\n\n| Quantization method                 | On the fly quantization | CPU | CUDA GPU | RoCm GPU (AMD) | Metal (Apple Silicon) | torch.compile() support | Number of bits | Supports fine-tuning (through PEFT) | Serializable with \ud83e\udd17 transformers | \ud83e\udd17 transformers support | Link to library                             |\n|-------------------------------------|-------------------------|-----|----------|----------------|-----------------------|-------------------------|----------------|-------------------------------------|--------------|------------------------|---------------------------------------------|\n| [AQLM](./aqlm)                                | \ud83d\udd34                       |  \ud83d\udfe2   |     \ud83d\udfe2     | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udfe2                      | 1 / 2          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/Vahe1994/AQLM            |\n| [AWQ](./awq) | \ud83d\udd34                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udfe2              | \ud83d\udd34                     | ?                       | 4              | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/casper-hansen/AutoAWQ    |\n| [bitsandbytes](./bitsandbytes)                        | \ud83d\udfe2                       | \ud83d\udd34   |     \ud83d\udfe2     | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udd34                       | 4 / 8          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/TimDettmers/bitsandbytes |\n| [EETQ](./eetq)                                | \ud83d\udfe2                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udd34                     | ?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4944f3a9-2e78-431f-bd2a-41a5b5bccde5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/overview", "title": "Quantization", "tokens": 1420, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4944f3a9-2e78-431f-bd2a-41a5b5bccde5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1420, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"db5af843-fb90-4c5d-8fdc-280c72d4586e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1420, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e75db12f43c89e8b8c8cf03490026c59210b8d9a659ff32dba2f66b319f926d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2847c2a4-2b4a-4e0d-8b5a-f237262ae5b0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/overview\", \"title\": \"Quantization\", \"tokens\": 1420, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d45b749676e765ee5de12ed3d608a8e2ddc0f81c7a2d31df6f6bcd76c0f27440\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1689, \"end_char_idx\": 5556, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "db5af843-fb90-4c5d-8fdc-280c72d4586e", "doc_id": "db5af843-fb90-4c5d-8fdc-280c72d4586e", "ref_doc_id": "db5af843-fb90-4c5d-8fdc-280c72d4586e", "chroma:document": "In short, supporting a wide range of quantization methods allows you to pick the best quantization method for your specific use case.\n\nUse the table below to help you decide which quantization method to use.\n\n| Quantization method                 | On the fly quantization | CPU | CUDA GPU | RoCm GPU (AMD) | Metal (Apple Silicon) | torch.compile() support | Number of bits | Supports fine-tuning (through PEFT) | Serializable with \ud83e\udd17 transformers | \ud83e\udd17 transformers support | Link to library                             |\n|-------------------------------------|-------------------------|-----|----------|----------------|-----------------------|-------------------------|----------------|-------------------------------------|--------------|------------------------|---------------------------------------------|\n| [AQLM](./aqlm)                                | \ud83d\udd34                       |  \ud83d\udfe2   |     \ud83d\udfe2     | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udfe2                      | 1 / 2          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/Vahe1994/AQLM            |\n| [AWQ](./awq) | \ud83d\udd34                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udfe2              | \ud83d\udd34                     | ?                       | 4              | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/casper-hansen/AutoAWQ    |\n| [bitsandbytes](./bitsandbytes)                        | \ud83d\udfe2                       | \ud83d\udd34   |     \ud83d\udfe2     | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udd34                       | 4 / 8          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/TimDettmers/bitsandbytes |\n| [EETQ](./eetq)                                | \ud83d\udfe2                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udd34                     | ?                       | 8              | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/NetEase-FuXi/EETQ        |\n| GGUF / GGML (llama.cpp)             | \ud83d\udfe2                       | \ud83d\udfe2   | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udfe2                     | \ud83d\udd34                       | 1 - 8          | \ud83d\udd34                                   | [See GGUF section](../gguf)                | [See GGUF section](../gguf)                      | https://github.com/ggerganov/llama.cpp      |\n| [GPTQ](./gptq)                                | \ud83d\udd34                       | \ud83d\udd34   | \ud83d\udfe2        | \ud83d\udfe2              | \ud83d\udd34                     | \ud83d\udd34                       | 2 - 3 - 4 - 8          | \ud83d\udfe2                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/AutoGPTQ/AutoGPTQ        |\n| [HQQ](./hqq)                                 | \ud83d\udfe2                       | \ud83d\udfe2    | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udd34                     | \ud83d\udfe2                       | 1 - 8          | \ud83d\udfe2                                   | \ud83d\udd34            | \ud83d\udfe2                      | https://github.com/mobiusml/hqq/            |\n| [Quanto](./quanto)                              | \ud83d\udfe2                       | \ud83d\udfe2   | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udfe2                     | \ud83d\udfe2                       | 2 / 4 / 8      | \ud83d\udd34                                   | \ud83d\udd34            | \ud83d\udfe2                      | https://github.com/huggingface/quanto       |\n| [FBGEMM_FP8](./fbgemm_fp8.md)                              | \ud83d\udfe2                       | \ud83d\udd34    | \ud83d\udfe2        | \ud83d\udd34              | \ud83d\udd34                      | \ud83d\udd34                        | 8      | \ud83d\udd34                                   | \ud83d\udfe2            | \ud83d\udfe2                      | https://github.com/pytorch/FBGEMM       |\n| [torchao](./torchao.md)                              | \ud83d\udfe2                       |     | \ud83d\udfe2        | \ud83d\udd34              | partial support (int4 weight only)       |                       | 4 / 8      |                                   | \ud83d\udfe2\ud83d\udd34           | \ud83d\udfe2                      | https://github.com/pytorch/ao       |"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3c3f2be9-0986-44cc-a7c8-8c38f3487af9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/fbgemm_fp8", "title": "FBGEMM FP8", "tokens": 661, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3c3f2be9-0986-44cc-a7c8-8c38f3487af9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/fbgemm_fp8\", \"title\": \"FBGEMM FP8\", \"tokens\": 661, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"620bbeac-6628-4b5f-8094-e0994aeaa9ef\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/fbgemm_fp8\", \"title\": \"FBGEMM FP8\", \"tokens\": 661, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"31eff12f3502d24517b20f40fa7c728cafd487fce4da742da0b93df258622468\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "620bbeac-6628-4b5f-8094-e0994aeaa9ef", "doc_id": "620bbeac-6628-4b5f-8094-e0994aeaa9ef", "ref_doc_id": "620bbeac-6628-4b5f-8094-e0994aeaa9ef", "chroma:document": "# FBGEMM FP8\n\nWith FBGEMM FP8 quantization method, you can quantize your model in FP8 (W8A8):\n- the weights will be quantized in 8bit (FP8) per channel\n- the activation will be quantized in 8bit (FP8) per token\n\nIt relies on the [FBGEMM](https://github.com/pytorch/FBGEMM) library which provides efficient low-precision general matrix multiplication for small batch sizes and support for accuracy-loss minimizing techniques such as row-wise quantization and outlier-aware quantization. \n\n> [!TIP]\n> You need a GPU with compute capability>=9 (e.g. H100) \n\nBefore you begin, make sure the following libraries are installed with their latest version:\n\n```bash\npip install --upgrade accelerate fbgemm-gpu torch\n```\n\nIf you are having issues with fbgemm-gpu and torch library, you might need to install the nighlty release. You can follow the instruction [here](https://pytorch.org/FBGEMM/fbgemm_gpu-development/InstallationInstructions.html#fbgemm-gpu-install-libraries:~:text=found%20here.-,Install%20the%20FBGEMM_GPU%20Package,-Install%20through%20PyTorch)\n\n\n```py\nfrom transformers import FbgemmFp8Config, AutoModelForCausalLM, AutoTokenizer\n\nmodel_name = \"meta-llama/Meta-Llama-3-8B\"\nquantization_config = FbgemmFp8Config()\nquantized_model = AutoModelForCausalLM.from_pretrained(model_name, device_map=\"auto\", quantization_config=quantization_config)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\ninput_text = \"What are we having for dinner?\"\ninput_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cuda\")\n\noutput = quantized_model.generate(**input_ids, max_new_tokens=10)\nprint(tokenizer.decode(output[0], skip_special_tokens=True))\n```\n\nA quantized model can be saved via \"saved_pretrained\" and be reused again via the \"from_pretrained\".\n\n```py\nquant_path = \"/path/to/save/quantized/model\"\nmodel.save_pretrained(quant_path)\nmodel = AutoModelForCausalLM.from_pretrained(quant_path, device_map=\"auto\")\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cce9d121-bf1c-4a42-ac64-42484b3a5363
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/optimum", "title": "Optimum", "tokens": 242, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cce9d121-bf1c-4a42-ac64-42484b3a5363\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/optimum\", \"title\": \"Optimum\", \"tokens\": 242, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b4afe632-d97f-42af-a780-71c93ab5315c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/optimum\", \"title\": \"Optimum\", \"tokens\": 242, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"92fb8e5a7999696bb2600dfc4a76f075bb402d115c7ea5e374469945cbb64e85\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 345, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b4afe632-d97f-42af-a780-71c93ab5315c", "doc_id": "b4afe632-d97f-42af-a780-71c93ab5315c", "ref_doc_id": "b4afe632-d97f-42af-a780-71c93ab5315c", "chroma:document": "# Optimum\n\nThe [Optimum](https://huggingface.co/docs/optimum/index) library supports quantization for Intel, Furiosa, ONNX Runtime, GPTQ, and lower-level PyTorch quantization functions. Consider using Optimum for quantization if you're using specific and optimized hardware like Intel CPUs, Furiosa NPUs or a model accelerator like ONNX Runtime."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3be9855c-98ff-4cbf-a76c-4c5bdbb07f23
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/quanto", "title": "Quanto", "tokens": 794, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3be9855c-98ff-4cbf-a76c-4c5bdbb07f23\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/quanto\", \"title\": \"Quanto\", \"tokens\": 794, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6764e390-454a-4856-a9da-4e52fbb7fdf1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/quanto\", \"title\": \"Quanto\", \"tokens\": 794, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec0c0e135462c2ee65b22053c3ce8571cf03c81eb523923823127e9efb031a89\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2583, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6764e390-454a-4856-a9da-4e52fbb7fdf1", "doc_id": "6764e390-454a-4856-a9da-4e52fbb7fdf1", "ref_doc_id": "6764e390-454a-4856-a9da-4e52fbb7fdf1", "chroma:document": "# Quanto\n\n<Tip>\n\nTry Quanto + transformers with this [notebook](https://colab.research.google.com/drive/16CXfVmtdQvciSh9BopZUDYcmXCDpvgrT?usp=sharing)!\n\n</Tip>\n\n\n[\ud83e\udd17 Quanto](https://github.com/huggingface/quanto) library is a versatile pytorch quantization toolkit. The quantization method used is the linear quantization. Quanto provides several unique features such as:\n\n- weights quantization (`float8`,`int8`,`int4`,`int2`)\n- activation quantization (`float8`,`int8`)\n- modality agnostic (e.g CV,LLM)\n- device agnostic (e.g CUDA,MPS,CPU)\n- compatibility with `torch.compile`\n- easy to add custom kernel for specific device\n- supports quantization aware training\n<!-- Add link to the blogpost -->\n\nBefore you begin, make sure the following libraries are installed:\n\n```bash\npip install quanto accelerate transformers\n```\n\nNow you can quantize a model by passing [`QuantoConfig`] object in the [`~PreTrainedModel.from_pretrained`] method. This works for any model in any modality, as long as it contains `torch.nn.Linear` layers. \n\nThe integration with transformers only supports weights quantization. For the more complex use case such as activation quantization, calibration and quantization aware training, you should use [quanto](https://github.com/huggingface/quanto) library instead. \n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, QuantoConfig\n\nmodel_id = \"facebook/opt-125m\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nquantization_config = QuantoConfig(weights=\"int8\")\nquantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"cuda:0\", quantization_config=quantization_config)\n```\n\nNote that serialization is not supported yet with transformers but it is coming soon! If you want to save the model, you can use quanto library instead.\n\nQuanto library uses linear quantization algorithm for quantization. Even though this is a basic quantization technique, we get very good results! Have a look at the following becnhmark (llama-2-7b on perplexity metric). You can find more benchamarks [here](https://github.com/huggingface/quanto/tree/main/bench/generation)\n\n<div class=\"flex gap-4\">\n  <div>\n    <img class=\"rounded-xl\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/quantization/NousResearch-Llama-2-7b-hf_Perplexity.png\" alt=\"llama-2-7b-quanto-perplexity\" />\n  </div>\n</div>\n\nThe library is versatible enough to be compatible with most PTQ optimization algorithms. The plan in the future is to integrate the most popular algorithms in the most seamless possible way (AWQ, Smoothquant)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 32b8625a-da0d-4fc5-a589-ae7d9d1ad746
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/contribute", "title": "Contribute new quantization method", "tokens": 1705, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"32b8625a-da0d-4fc5-a589-ae7d9d1ad746\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"04ba2156-3f37-4dfa-89f7-846815161617\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53fbf61f9596f0cbe6ceca6e165cb2aa79c2308dae7bcd4afac9e225c76e7e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ef9e4a60-5f1e-4c18-9e56-7c3b0b8e1546\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"923c92d9fac385259515586e3549ce9638141d1d78921a2c3ce003e5c3e81172\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3192, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "04ba2156-3f37-4dfa-89f7-846815161617", "doc_id": "04ba2156-3f37-4dfa-89f7-846815161617", "ref_doc_id": "04ba2156-3f37-4dfa-89f7-846815161617", "chroma:document": "# Contribute new quantization method\n\nTransformers supports and integrates many quantization methods such as QLoRA, GPTQ, LLM.int8, and AWQ. However, there are other quantization approaches that are not yet integrated. To make adding and using these quantization methods with Transformers models easier, you should use the [`HfQuantizer`] class. The [`HfQuantizer`] is designed as an internal helper class for adding a quantization method instead of something you apply to every PyTorch module.\n\nThis guide will show you how to integrate a new quantization method with the [`HfQuantizer`] class.\n\n## Requirements\n\nBefore integrating a new quantization method into Transformers, ensure the method you are trying to add meets the following prerequisites. Only quantization methods that can be run with PyTorch modules are currently supported.\n\n- The quantization method is available through a Python package that is pip-installable by anyone (it is also fine if you can only install the package from source). Ideally, pre-compiled kernels are included in the pip package.\n- The method can run on commonly-used hardware (CPU, GPU, ...).\n- The method is wrapped in a `nn.Module` (e.g., `Linear8bitLt`, `Linear4bit`), and the quantized linear layer should have the following definition:\n\n```py\nclass Linear4bit(nn.Module):\n    def __init__(self, ...):\n        ...\n    \n    def forward(self, x):\n        return my_4bit_kernel(x, self.weight, self.bias)\n```\n\nThis way, Transformers models can be easily quantized by replacing some instances of `nn.Linear` with a target class.\n\n- The quantization method should be serializable. You can save the quantized weights locally or push them to the Hub.\n- Make sure the package that contains the quantization kernels/primitive is stable (no frequent breaking changes).\n\nFor some quantization methods, they may require \"pre-quantizing\" the models through data calibration (e.g., AWQ). In this case, we prefer to only support inference in Transformers and let the third-party library maintained by the ML community deal with the model quantization itself.\n\n## Build a new HFQuantizer class\n\n1. Create a new quantization config class inside [src/transformers/utils/quantization_config.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/utils/quantization_config.py) and make sure to expose the new quantization config inside Transformers main `init` by adding it to the [`_import_structure`](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/__init__.py#L1088) object of [src/transformers/__init__.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/__init__.py).\n\n2. Create a new file inside [src/transformers/quantizers/](https://github.com/huggingface/transformers/tree/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers) named `quantizer_your_method.py`, and make it inherit from [src/transformers/quantizers/base.py::HfQuantizer](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/base.py#L28)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ef9e4a60-5f1e-4c18-9e56-7c3b0b8e1546
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/contribute", "title": "Contribute new quantization method", "tokens": 1705, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ef9e4a60-5f1e-4c18-9e56-7c3b0b8e1546\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"04ba2156-3f37-4dfa-89f7-846815161617\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53fbf61f9596f0cbe6ceca6e165cb2aa79c2308dae7bcd4afac9e225c76e7e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"32b8625a-da0d-4fc5-a589-ae7d9d1ad746\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b56c88b85a7075ba1efb1648cb6feb05210f57ce70218cbd7c91aa08f32d558a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e52acd8d-a4d3-42e4-ae4f-04eea45f2708\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"17ea5e59814c4736a2335e1e30bee769f99f0b0618b8fc13165b59164eb7f821\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1805, \"end_char_idx\": 4854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "04ba2156-3f37-4dfa-89f7-846815161617", "doc_id": "04ba2156-3f37-4dfa-89f7-846815161617", "ref_doc_id": "04ba2156-3f37-4dfa-89f7-846815161617", "chroma:document": "For some quantization methods, they may require \"pre-quantizing\" the models through data calibration (e.g., AWQ). In this case, we prefer to only support inference in Transformers and let the third-party library maintained by the ML community deal with the model quantization itself.\n\n## Build a new HFQuantizer class\n\n1. Create a new quantization config class inside [src/transformers/utils/quantization_config.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/utils/quantization_config.py) and make sure to expose the new quantization config inside Transformers main `init` by adding it to the [`_import_structure`](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/__init__.py#L1088) object of [src/transformers/__init__.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/__init__.py).\n\n2. Create a new file inside [src/transformers/quantizers/](https://github.com/huggingface/transformers/tree/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers) named `quantizer_your_method.py`, and make it inherit from [src/transformers/quantizers/base.py::HfQuantizer](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/base.py#L28). Make sure to add the new quantizer and quantization config in the quantization auto-mapping in [src/transformers/quantizers/auto.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/auto.py).\n\n3. Define the following class attributes/property methods for your quantization method:\n\n* `requires_calibration`: Whether the quantization method requires a data calibration process. If set to `True`, you can only support inference (with quantized weights) and not inference and quantization.\n* `required_packages`: A list of strings of the required packages to use the quantized weights. You might need to define some new utility methods such as `is_auto_awq_available` in [transformers/src/utils/import_utils.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/utils/import_utils.py).\n* `requires_parameters_quantization`: Only required if your quantization method requires extra attention to the underlying `nn.Parameter` object. For example, bitsandbytes uses `Params4bit` and `Int8Param`, which requires some extra attention when quantizing the model. Most of the recent quantization method packs int2/int4 weights inside `torch.uint8` weights, so this flag should not be really required (set to `False` by default).\n* `is_serializable`: A property method to determine whether the method is serializable or not.\n* `is_trainable`:  A property method to determine whether you can fine-tune models on top of the quantization method (with or without PEFT approaches).\n\n4. Write the `validate_environment` and `update_torch_dtype` methods."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e52acd8d-a4d3-42e4-ae4f-04eea45f2708
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/contribute", "title": "Contribute new quantization method", "tokens": 1705, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e52acd8d-a4d3-42e4-ae4f-04eea45f2708\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"04ba2156-3f37-4dfa-89f7-846815161617\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53fbf61f9596f0cbe6ceca6e165cb2aa79c2308dae7bcd4afac9e225c76e7e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ef9e4a60-5f1e-4c18-9e56-7c3b0b8e1546\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61ce5062dfb13c5b7aa2824d68b6726c30d0d2983420ab5b8b5081c19225dcfb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"37d846ab-64d6-41ed-a3c7-28bba99a3eaf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d9c9e8bcb461c14adc0524f1d6cde9bb70541d415e8e64baf603d27ee423a466\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3193, \"end_char_idx\": 6548, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "04ba2156-3f37-4dfa-89f7-846815161617", "doc_id": "04ba2156-3f37-4dfa-89f7-846815161617", "ref_doc_id": "04ba2156-3f37-4dfa-89f7-846815161617", "chroma:document": "Make sure to add the new quantizer and quantization config in the quantization auto-mapping in [src/transformers/quantizers/auto.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/auto.py).\n\n3. Define the following class attributes/property methods for your quantization method:\n\n* `requires_calibration`: Whether the quantization method requires a data calibration process. If set to `True`, you can only support inference (with quantized weights) and not inference and quantization.\n* `required_packages`: A list of strings of the required packages to use the quantized weights. You might need to define some new utility methods such as `is_auto_awq_available` in [transformers/src/utils/import_utils.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/utils/import_utils.py).\n* `requires_parameters_quantization`: Only required if your quantization method requires extra attention to the underlying `nn.Parameter` object. For example, bitsandbytes uses `Params4bit` and `Int8Param`, which requires some extra attention when quantizing the model. Most of the recent quantization method packs int2/int4 weights inside `torch.uint8` weights, so this flag should not be really required (set to `False` by default).\n* `is_serializable`: A property method to determine whether the method is serializable or not.\n* `is_trainable`:  A property method to determine whether you can fine-tune models on top of the quantization method (with or without PEFT approaches).\n\n4. Write the `validate_environment` and `update_torch_dtype` methods. These methods are called before creating the quantized model to ensure users use the right configuration. You can have a look at how this is done on other quantizers.\n\n5. Write the `_process_model_before_weight_loading` method. In Transformers, the quantized models are initialized first on the `\"meta\"` device before loading the weights. This means the `_process_model_before_weight_loading` method takes care of manipulating the model skeleton to replace some modules (e.g., `nn.Linear`) with the target modules (quantization modules). You can define a module replacement logic or any other utility method by creating a new file in [transformers/src/integrations/](https://github.com/huggingface/transformers/tree/abbffc4525566a48a9733639797c812301218b83/src/transformers/integrations) and exposing the relevant methods in that folder's `__init__.py` file. The best starting point would be to have a look at another quantization methods such as [quantizer_awq.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/quantizer_awq.py).\n\n6. Write the `_process_model_after_weight_loading` method. This method enables implementing additional features that require manipulating the model after loading the weights.\n\n7. Document everything! Make sure your quantization method is documented by adding a new file under `docs/source/en/quantization` and adding a new row in the table in `docs/source/en/quantization/overview.md`.\n\n8. Add tests! You should add tests by first adding the package in our nightly Dockerfile inside `docker/transformers-quantization-latest-gpu` and then adding a new test file in `tests/quantization/xxx`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 37d846ab-64d6-41ed-a3c7-28bba99a3eaf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/contribute", "title": "Contribute new quantization method", "tokens": 1705, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"37d846ab-64d6-41ed-a3c7-28bba99a3eaf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"04ba2156-3f37-4dfa-89f7-846815161617\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53fbf61f9596f0cbe6ceca6e165cb2aa79c2308dae7bcd4afac9e225c76e7e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e52acd8d-a4d3-42e4-ae4f-04eea45f2708\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/contribute\", \"title\": \"Contribute new quantization method\", \"tokens\": 1705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"efcc377ade78cc2272044bb17ef71350e1a12f77e60fc341596e19e479027083\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4855, \"end_char_idx\": 6625, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "04ba2156-3f37-4dfa-89f7-846815161617", "doc_id": "04ba2156-3f37-4dfa-89f7-846815161617", "ref_doc_id": "04ba2156-3f37-4dfa-89f7-846815161617", "chroma:document": "These methods are called before creating the quantized model to ensure users use the right configuration. You can have a look at how this is done on other quantizers.\n\n5. Write the `_process_model_before_weight_loading` method. In Transformers, the quantized models are initialized first on the `\"meta\"` device before loading the weights. This means the `_process_model_before_weight_loading` method takes care of manipulating the model skeleton to replace some modules (e.g., `nn.Linear`) with the target modules (quantization modules). You can define a module replacement logic or any other utility method by creating a new file in [transformers/src/integrations/](https://github.com/huggingface/transformers/tree/abbffc4525566a48a9733639797c812301218b83/src/transformers/integrations) and exposing the relevant methods in that folder's `__init__.py` file. The best starting point would be to have a look at another quantization methods such as [quantizer_awq.py](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/quantizers/quantizer_awq.py).\n\n6. Write the `_process_model_after_weight_loading` method. This method enables implementing additional features that require manipulating the model after loading the weights.\n\n7. Document everything! Make sure your quantization method is documented by adding a new file under `docs/source/en/quantization` and adding a new row in the table in `docs/source/en/quantization/overview.md`.\n\n8. Add tests! You should add tests by first adding the package in our nightly Dockerfile inside `docker/transformers-quantization-latest-gpu` and then adding a new test file in `tests/quantization/xxx`. Feel free to check out how it is implemented for other quantization methods."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8f311452-1d98-4a37-a35a-2c23def8c633
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/hqq", "title": "HQQ", "tokens": 738, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8f311452-1d98-4a37-a35a-2c23def8c633\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/hqq\", \"title\": \"HQQ\", \"tokens\": 738, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c06e14a-c575-4094-8c2a-c3dc44bef7a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/hqq\", \"title\": \"HQQ\", \"tokens\": 738, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e7d8fd133e251cc5c8726d994b0d687606ec32f7309c727ecdf1bcb4bf4b1d20\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2161, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c06e14a-c575-4094-8c2a-c3dc44bef7a0", "doc_id": "0c06e14a-c575-4094-8c2a-c3dc44bef7a0", "ref_doc_id": "0c06e14a-c575-4094-8c2a-c3dc44bef7a0", "chroma:document": "# HQQ \n\nHalf-Quadratic Quantization (HQQ) implements on-the-fly quantization via fast robust optimization. It doesn't require calibration data and can be used to quantize any model.  \nPlease refer to the <a href=\"https://github.com/mobiusml/hqq/\">official package</a> for more details.\n\nFor installation, we recommend you use the following approach to get the latest version and build its corresponding CUDA kernels:\n```\npip install hqq\n```\n\nTo quantize a model, you need to create an [`HqqConfig`]. There are two ways of doing it:\n``` Python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# Method 1: all linear layers will use the same quantization config\nquant_config  = HqqConfig(nbits=8, group_size=64, quant_zero=False, quant_scale=False, axis=0) #axis=0 is used by default\n```\n\n``` Python\n# Method 2: each linear layer with the same tag will use a dedicated quantization config\nq4_config = {'nbits':4, 'group_size':64, 'quant_zero':False, 'quant_scale':False}\nq3_config = {'nbits':3, 'group_size':32, 'quant_zero':False, 'quant_scale':False}\nquant_config  = HqqConfig(dynamic_config={\n  'self_attn.q_proj':q4_config,\n  'self_attn.k_proj':q4_config,\n  'self_attn.v_proj':q4_config,\n  'self_attn.o_proj':q4_config,\n\n  'mlp.gate_proj':q3_config,\n  'mlp.up_proj'  :q3_config,\n  'mlp.down_proj':q3_config,\n})\n```\n\nThe second approach is especially interesting for quantizing Mixture-of-Experts (MoEs) because the experts are less affected by lower quantization settings.\n\n\nThen you simply quantize the model as follows\n``` Python\nmodel = transformers.AutoModelForCausalLM.from_pretrained(\n    model_id, \n    torch_dtype=torch.float16, \n    device_map=\"cuda\", \n    quantization_config=quant_config\n)\n```\n\n## Optimized Runtime\n\nHQQ supports various backends, including pure Pytorch and custom dequantization CUDA kernels. These backends are suitable for older gpus and peft/QLoRA training.\nFor faster inference, HQQ supports 4-bit fused kernels (TorchAO and Marlin), reaching up to 200 tokens/sec on a single 4090.\nFor more details on how to use the backends, please refer to https://github.com/mobiusml/hqq/?tab=readme-ov-file#backend"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 792b2e9d-9ac1-4ad7-8584-b23cdb5741db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quantization/aqlm", "title": "AQLM", "tokens": 838, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"792b2e9d-9ac1-4ad7-8584-b23cdb5741db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/aqlm\", \"title\": \"AQLM\", \"tokens\": 838, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9d564ed4-9509-403b-9df1-b695b08c2b76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quantization/aqlm\", \"title\": \"AQLM\", \"tokens\": 838, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3cbb09e29701a2808aec57be44154966ecbcded946a32cacee911da3a61bb79\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2512, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9d564ed4-9509-403b-9df1-b695b08c2b76", "doc_id": "9d564ed4-9509-403b-9df1-b695b08c2b76", "ref_doc_id": "9d564ed4-9509-403b-9df1-b695b08c2b76", "chroma:document": "# AQLM\n\n> [!TIP]\n> Try AQLM on [Google Colab](https://colab.research.google.com/drive/1-xZmBRXT5Fm3Ghn4Mwa2KRypORXb855X?usp=sharing)!\n\nAdditive Quantization of Language Models ([AQLM](https://arxiv.org/abs/2401.06118)) is a Large Language Models compression method. It quantizes multiple weights together and take advantage of interdependencies between them. AQLM represents groups of 8-16 weights as a sum of multiple vector codes.\n\nInference support for AQLM is realised in the `aqlm` library. Make sure to install it to run the models (note aqlm works only with python>=3.10):\n```bash\npip install aqlm[gpu,cpu]\n```\n\nThe library provides efficient kernels for both GPU and CPU inference and training.\n\nThe instructions on how to quantize models yourself, as well as all the relevant code can be found in the corresponding GitHub [repository](https://github.com/Vahe1994/AQLM). To run AQLM models simply load a model that has been quantized with AQLM:\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nquantized_model = AutoModelForCausalLM.from_pretrained(\n    \"ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf\",\n    torch_dtype=\"auto\", \n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf\")\n```\n\n## PEFT\n\nStarting with version `aqlm 1.0.2`, AQLM supports Parameter-Efficient Fine-Tuning in a form of [LoRA](https://huggingface.co/docs/peft/package_reference/lora) integrated into the [PEFT](https://huggingface.co/blog/peft) library.\n\n## AQLM configurations\n\nAQLM quantization setups vary mainly on the number of codebooks used as well as codebook sizes in bits. The most popular setups, as well as inference kernels they support are:\n \n| Kernel | Number of codebooks | Codebook size, bits | Notation | Accuracy | Speedup     | Fast GPU inference | Fast CPU inference |\n|---|---------------------|---------------------|----------|-------------|-------------|--------------------|--------------------|\n| Triton | K                   | N                  | KxN     | -        | Up to ~0.7x | \u2705                  | \u274c                  |\n| CUDA | 1                   | 16                  | 1x16     | Best        | Up to ~1.3x | \u2705                  | \u274c                  |\n| CUDA | 2                   | 8                   | 2x8      | OK          | Up to ~3.0x | \u2705                  | \u274c                  |\n| Numba | K                   | 8                   | Kx8      | Good        | Up to ~4.0x | \u274c                  | \u2705                  |"}


# collections

id: 621deaf2-9359-4ac4-ada0-047176703e13
name: chroma-db-transformers
database_id: 00000000-0000-0000-0000-000000000000
config_json_str:
{"hnsw_configuration": {"space": "l2", "ef_construction": 100, "ef_search": 10, "num_threads": 8, "M": 16, "resize_factor": 1.2, "batch_size": 100, "sync_threshold": 1000, "_type": "HNSWConfigurationInternal"}, "_type": "CollectionConfigurationInternal"}


# embedding_metadata

key: _node_content
string_value:
{"id_": "6741a682-7447-4fb1-8ad9-1177267959a0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "278b5932-2ca4-4f7f-97ab-6fe2ba2cda07", "node_type": "1", "metadata": {}, "hash": "7b6e926217884aa4d45d293ae4066739adfb48fddce666f2b7ac2b49e5479503", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2903, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Preprocess

[[open-in-colab]]

Before you can train a model on a dataset, it needs to be preprocessed into the expected model input format. Whether your data is text, images, or audio, they need to be converted and assembled into batches of tensors. 🤗 Transformers provides a set of preprocessing classes to help prepare your data for the model. In this tutorial, you'll learn that for:

* Text, use a [Tokenizer](./main_classes/tokenizer) to convert text into a sequence of tokens, create a numerical representation of the tokens, and assemble them into tensors.
* Speech and audio, use a [Feature extractor](./main_classes/feature_extractor) to extract sequential features from audio waveforms and convert them into tensors.
* Image inputs use a [ImageProcessor](./main_classes/image_processor) to convert images into tensors.
* Multimodal inputs, use a [Processor](./main_classes/processors) to combine a tokenizer and a feature extractor or image processor.

<Tip>

`AutoProcessor` **always** works and automatically chooses the correct class for the model you're using, whether you're using a tokenizer, image processor, feature extractor or processor.

</Tip>

Before you begin, install 🤗 Datasets so you can load some datasets to experiment with:

```bash
pip install datasets
```

## Natural Language Processing

<Youtube id="Yffk5aydLzg"/>

The main tool for preprocessing textual data is a [tokenizer](main_classes/tokenizer). A tokenizer splits text into *tokens* according to a set of rules. The tokens are converted into numbers and then tensors, which become the model inputs. Any additional inputs required by the model are added by the tokenizer.

<Tip>

If you plan on using a pretrained model, it's important to use the associated pretrained tokenizer. This ensures the text is split the same way as the pretraining corpus, and uses the same corresponding tokens-to-index (usually referred to as the *vocab*) during pretraining.

</Tip>

Get started by loading a pretrained tokenizer with the [`AutoTokenizer.from_pretrained`] method. This downloads the *vocab* a model was pretrained with:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")
```

Then pass your text to the tokenizer:

```py
>>> encoded_input = tokenizer("Do not meddle in the affairs of wizards, for they are subtle and quick to anger.")
>>> print(encoded_input)
{'input_ids': [101, 2079, 2025, 19960, 10362, 1999, 1996, 3821, 1997, 16657, 1010, 2005, 2027, 2024, 11259, 1998, 4248, 2000, 4963, 1012, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

The tokenizer returns a dictionary with three important items:

* [input_ids](glossary#input-ids) are the indices corresponding to each token in the sentence.


key: _node_content
string_value:
{"id_": "278b5932-2ca4-4f7f-97ab-6fe2ba2cda07", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6741a682-7447-4fb1-8ad9-1177267959a0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "cb33f20b07ca1a9f80777824548579572d967acce85051e22799c63e763bccb9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d2e8ad1-c810-44d1-a31f-4a61d4f474d1", "node_type": "1", "metadata": {}, "hash": "8336be552eec8e5c8f393668167fc2d4ec77acc072931d5e0e258f60c419c4c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1773, "end_char_idx": 3814, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This ensures the text is split the same way as the pretraining corpus, and uses the same corresponding tokens-to-index (usually referred to as the *vocab*) during pretraining.

</Tip>

Get started by loading a pretrained tokenizer with the [`AutoTokenizer.from_pretrained`] method. This downloads the *vocab* a model was pretrained with:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")
```

Then pass your text to the tokenizer:

```py
>>> encoded_input = tokenizer("Do not meddle in the affairs of wizards, for they are subtle and quick to anger.")
>>> print(encoded_input)
{'input_ids': [101, 2079, 2025, 19960, 10362, 1999, 1996, 3821, 1997, 16657, 1010, 2005, 2027, 2024, 11259, 1998, 4248, 2000, 4963, 1012, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

The tokenizer returns a dictionary with three important items:

* [input_ids](glossary#input-ids) are the indices corresponding to each token in the sentence.
* [attention_mask](glossary#attention-mask) indicates whether a token should be attended to or not.
* [token_type_ids](glossary#token-type-ids) identifies which sequence a token belongs to when there is more than one sequence.

Return your input by decoding the `input_ids`:

```py
>>> tokenizer.decode(encoded_input["input_ids"])
'[CLS] Do not meddle in the affairs of wizards, for they are subtle and quick to anger. [SEP]'
```

As you can see, the tokenizer added two special tokens - `CLS` and `SEP` (classifier and separator) - to the sentence. Not all models need
special tokens, but if they do, the tokenizer automatically adds them for you.

If there are several sentences you want to preprocess, pass them as a list to the tokenizer:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "9d2e8ad1-c810-44d1-a31f-4a61d4f474d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "278b5932-2ca4-4f7f-97ab-6fe2ba2cda07", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "82dae28fc3d6d06948bcc12f0153a5e5ba9366d4de83ca10c306720f9c9ff264", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bc261b6d-bacb-4157-83c5-9b6a4352c005", "node_type": "1", "metadata": {}, "hash": "bd9f6313951e6714e51bc15e1f90fac0aac4da7a8e937e5b66bb62961712c8b5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2904, "end_char_idx": 4970, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* [attention_mask](glossary#attention-mask) indicates whether a token should be attended to or not.
* [token_type_ids](glossary#token-type-ids) identifies which sequence a token belongs to when there is more than one sequence.

Return your input by decoding the `input_ids`:

```py
>>> tokenizer.decode(encoded_input["input_ids"])
'[CLS] Do not meddle in the affairs of wizards, for they are subtle and quick to anger. [SEP]'
```

As you can see, the tokenizer added two special tokens - `CLS` and `SEP` (classifier and separator) - to the sentence. Not all models need
special tokens, but if they do, the tokenizer automatically adds them for you.

If there are several sentences you want to preprocess, pass them as a list to the tokenizer:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_inputs = tokenizer(batch_sentences)
>>> print(encoded_inputs)
{'input_ids': [[101, 1252, 1184, 1164, 1248, 6462, 136, 102],
               [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
               [101, 1327, 1164, 5450, 23434, 136, 102]],
 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0]],
 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1]]}
```

### Pad

Sentences aren't always the same length which can be an issue because tensors, the model inputs, need to have a uniform shape. Padding is a strategy for ensuring tensors are rectangular by adding a special *padding token* to shorter sentences.

Set the `padding` parameter to `True` to pad the shorter sequences in the batch to match the longest sequence:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "bc261b6d-bacb-4157-83c5-9b6a4352c005", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9d2e8ad1-c810-44d1-a31f-4a61d4f474d1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "7724906a002dc54e0aeb7f8ce7e2b11bb1633192a02d1b3893a62a537ccd5151", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "86f9fbca-c46b-4d9f-bef5-93c9aadad1f4", "node_type": "1", "metadata": {}, "hash": "3250eb6b2bb63bb28fbd62d8853a0e87b7f5aa26e1ebcc850c8a2524edb5198e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4574, "end_char_idx": 6283, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Padding is a strategy for ensuring tensors are rectangular by adding a special *padding token* to shorter sentences.

Set the `padding` parameter to `True` to pad the shorter sequences in the batch to match the longest sequence:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True)
>>> print(encoded_input)
{'input_ids': [[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
               [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
               [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]],
 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
                    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]]}
```

The first and third sentences are now padded with `0`'s because they are shorter.

### Truncation

On the other end of the spectrum, sometimes a sequence may be too long for a model to handle. In this case, you'll need to truncate the sequence to a shorter length.

Set the `truncation` parameter to `True` to truncate a sequence to the maximum length accepted by the model:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "86f9fbca-c46b-4d9f-bef5-93c9aadad1f4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc261b6d-bacb-4157-83c5-9b6a4352c005", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "1505b58ef2d673e96cfe0eecea44a21fc3469d6562e8197937d906ba622d23d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d0b7dbda-7221-4837-a19e-9d083c4cea81", "node_type": "1", "metadata": {}, "hash": "f34879afe7db6f923540c8fe90820cb6a8b79ffca3281adadf5751f16a091d58", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5824, "end_char_idx": 7603, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Truncation

On the other end of the spectrum, sometimes a sequence may be too long for a model to handle. In this case, you'll need to truncate the sequence to a shorter length.

Set the `truncation` parameter to `True` to truncate a sequence to the maximum length accepted by the model:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True, truncation=True)
>>> print(encoded_input)
{'input_ids': [[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
               [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
               [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]],
 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
                    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]]}
```

<Tip>

Check out the [Padding and truncation](./pad_truncation) concept guide to learn more different padding and truncation arguments.

</Tip>

### Build tensors

Finally, you want the tokenizer to return the actual tensors that get fed to the model.

Set the `return_tensors` parameter to either `pt` for PyTorch, or `tf` for TensorFlow:

<frameworkcontent>
<pt>

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "d0b7dbda-7221-4837-a19e-9d083c4cea81", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "86f9fbca-c46b-4d9f-bef5-93c9aadad1f4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "bee24609fcad865fd1e0cd04c39f87a6448e9eb57ee230b0f18342df56df06c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bf94f656-52b0-42d7-9f5c-ea530911f968", "node_type": "1", "metadata": {}, "hash": "e8b58e4f6ef3714c1b8d88ab0ceb04951452251409f5513703a567bbcb65143b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7208, "end_char_idx": 8654, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

### Build tensors

Finally, you want the tokenizer to return the actual tensors that get fed to the model.

Set the `return_tensors` parameter to either `pt` for PyTorch, or `tf` for TensorFlow:

<frameworkcontent>
<pt>

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors="pt")
>>> print(encoded_input)
{'input_ids': tensor([[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
                      [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
                      [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]]),
 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                           [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                           [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]),
 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
                           [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                           [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]])}
```
</pt>
<tf>
```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "bf94f656-52b0-42d7-9f5c-ea530911f968", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d0b7dbda-7221-4837-a19e-9d083c4cea81", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "475eb98a98a46cefa9f739543358037cf734ac45598a1fa9e5a79ec7b6366342", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "616316e4-e409-4961-815c-e090ccf29c6a", "node_type": "1", "metadata": {}, "hash": "8318013a55868ead22fceda3515200ae8857fa4a7a892b687c376aa66ed8975a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8559, "end_char_idx": 10249, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors="tf")
>>> print(encoded_input)
{'input_ids': <tf.Tensor: shape=(2, 9), dtype=int32, numpy=
array([[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
       [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
       [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]],
      dtype=int32)>,
 'token_type_ids': <tf.Tensor: shape=(2, 9), dtype=int32, numpy=
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
 'attention_mask': <tf.Tensor: shape=(2, 9), dtype=int32, numpy=
array([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)>}
```
</tf>
</frameworkcontent>

<Tip>
Different pipelines support tokenizer arguments in their `__call__()` differently. `text-2-text-generation` pipelines support (i.e. pass on)
only `truncation`. `text-generation` pipelines support `max_length`, `truncation`, `padding` and `add_special_tokens`. 
In `fill-mask` pipelines, tokenizer arguments can be passed in the `tokenizer_kwargs` argument (dictionary).
</Tip>

## Audio

For audio tasks, you'll need a [feature extractor](main_classes/feature_extractor) to prepare your dataset for the model. The feature extractor is designed to extract features from raw audio data, and convert them into tensors.


key: _node_content
string_value:
{"id_": "616316e4-e409-4961-815c-e090ccf29c6a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bf94f656-52b0-42d7-9f5c-ea530911f968", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "f50103c82d1802a67e0d4d5b02df3542e0a16aa10831d0e848bbd6c073431316", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "be977fd8-b4a2-49ac-b238-028d94c7ea1f", "node_type": "1", "metadata": {}, "hash": "4b45e3f324520fefa6e70718e76fad19d056612febb742f115eaa857818f8e68", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9717, "end_char_idx": 12006, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
`text-2-text-generation` pipelines support (i.e. pass on)
only `truncation`. `text-generation` pipelines support `max_length`, `truncation`, `padding` and `add_special_tokens`. 
In `fill-mask` pipelines, tokenizer arguments can be passed in the `tokenizer_kwargs` argument (dictionary).
</Tip>

## Audio

For audio tasks, you'll need a [feature extractor](main_classes/feature_extractor) to prepare your dataset for the model. The feature extractor is designed to extract features from raw audio data, and convert them into tensors.

Load the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use a feature extractor with audio datasets:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```

Access the first element of the `audio` column to take a look at the input. Calling the `audio` column automatically loads and resamples the audio file:

```py
>>> dataset[0]["audio"]
{'array': array([ 0.        ,  0.00024414, -0.00024414, ..., -0.00024414,
         0.        ,  0.        ], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 8000}
```

This returns three items:

* `array` is the speech signal loaded - and potentially resampled - as a 1D array.
* `path` points to the location of the audio file.
* `sampling_rate` refers to how many data points in the speech signal are measured per second.

For this tutorial, you'll use the [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) model. Take a look at the model card, and you'll learn Wav2Vec2 is pretrained on 16kHz sampled speech audio. It is important your audio data's sampling rate matches the sampling rate of the dataset used to pretrain the model. If your data's sampling rate isn't the same, then you need to resample your data.

1. Use 🤗 Datasets' [`~datasets.Dataset.cast_column`] method to upsample the sampling rate to 16kHz:

```py
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))
```

2.


key: _node_content
string_value:
{"id_": "be977fd8-b4a2-49ac-b238-028d94c7ea1f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "616316e4-e409-4961-815c-e090ccf29c6a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eae12348a62d278c14a344b2dae0214b11cd38c6ee262077711804e2e917ebff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d567f0c7-6f4b-4f95-b6ef-ac766d75037e", "node_type": "1", "metadata": {}, "hash": "0ede54a9d0ee9936ac77410e7cd1850768f9353bcf60dcc8f7575615d2e6e736", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10720, "end_char_idx": 13097, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Calling the `audio` column automatically loads and resamples the audio file:

```py
>>> dataset[0]["audio"]
{'array': array([ 0.        ,  0.00024414, -0.00024414, ..., -0.00024414,
         0.        ,  0.        ], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 8000}
```

This returns three items:

* `array` is the speech signal loaded - and potentially resampled - as a 1D array.
* `path` points to the location of the audio file.
* `sampling_rate` refers to how many data points in the speech signal are measured per second.

For this tutorial, you'll use the [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) model. Take a look at the model card, and you'll learn Wav2Vec2 is pretrained on 16kHz sampled speech audio. It is important your audio data's sampling rate matches the sampling rate of the dataset used to pretrain the model. If your data's sampling rate isn't the same, then you need to resample your data.

1. Use 🤗 Datasets' [`~datasets.Dataset.cast_column`] method to upsample the sampling rate to 16kHz:

```py
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))
```

2. Call the `audio` column again to resample the audio file:

```py
>>> dataset[0]["audio"]
{'array': array([ 2.3443763e-05,  2.1729663e-04,  2.2145823e-04, ...,
         3.8356509e-05, -7.3497440e-06, -2.1754686e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 16000}
```

Next, load a feature extractor to normalize and pad the input. When padding textual data, a `0` is added for shorter sequences. The same idea applies to audio data. The feature extractor adds a `0` - interpreted as silence - to `array`.

Load the feature extractor with [`AutoFeatureExtractor.from_pretrained`]:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
```

Pass the audio `array` to the feature extractor. We also recommend adding the `sampling_rate` argument in the feature extractor in order to better debug any silent errors that may occur.


key: _node_content
string_value:
{"id_": "d567f0c7-6f4b-4f95-b6ef-ac766d75037e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "be977fd8-b4a2-49ac-b238-028d94c7ea1f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "a38cbad9766cb401559c034fef6a25d91aed93818efca515ee56c30d7e1de8bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "851c446c-847e-4add-b90c-5d89e0e0a4bf", "node_type": "1", "metadata": {}, "hash": "ce4b82e7e4557e91950b591b48945e9f46d177f24a96da01936e1f6dbe87dfb7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11737, "end_char_idx": 14442, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If your data's sampling rate isn't the same, then you need to resample your data.

1. Use 🤗 Datasets' [`~datasets.Dataset.cast_column`] method to upsample the sampling rate to 16kHz:

```py
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))
```

2. Call the `audio` column again to resample the audio file:

```py
>>> dataset[0]["audio"]
{'array': array([ 2.3443763e-05,  2.1729663e-04,  2.2145823e-04, ...,
         3.8356509e-05, -7.3497440e-06, -2.1754686e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 16000}
```

Next, load a feature extractor to normalize and pad the input. When padding textual data, a `0` is added for shorter sequences. The same idea applies to audio data. The feature extractor adds a `0` - interpreted as silence - to `array`.

Load the feature extractor with [`AutoFeatureExtractor.from_pretrained`]:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
```

Pass the audio `array` to the feature extractor. We also recommend adding the `sampling_rate` argument in the feature extractor in order to better debug any silent errors that may occur.

```py
>>> audio_input = [dataset[0]["audio"]["array"]]
>>> feature_extractor(audio_input, sampling_rate=16000)
{'input_values': [array([ 3.8106556e-04,  2.7506407e-03,  2.8015103e-03, ...,
        5.6335266e-04,  4.6588284e-06, -1.7142107e-04], dtype=float32)]}
```

Just like the tokenizer, you can apply padding or truncation to handle variable sequences in a batch. Take a look at the sequence length of these two audio samples:

```py
>>> dataset[0]["audio"]["array"].shape
(173398,)

>>> dataset[1]["audio"]["array"].shape
(106496,)
```

Create a function to preprocess the dataset so the audio samples are the same lengths. Specify a maximum sample length, and the feature extractor will either pad or truncate the sequences to match it:

```py
>>> def preprocess_function(examples):
...     audio_arrays = [x["array"] for x in examples["audio"]]
...     inputs = feature_extractor(
...         audio_arrays,
...         sampling_rate=16000,
...         padding=True,
...         max_length=100000,
...         truncation=True,
...     )
...     return inputs
```

Apply the `preprocess_function` to the first few examples in the dataset:

```py
>>> processed_dataset = preprocess_function(dataset[:5])
```

The sample lengths are now the same and match the specified maximum length. You can pass your processed dataset to the model now!


key: _node_content
string_value:
{"id_": "851c446c-847e-4add-b90c-5d89e0e0a4bf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d567f0c7-6f4b-4f95-b6ef-ac766d75037e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "d9cb780779a56717491def3d88e478e65eb7ff6895cd3d9c7b68beb69f32f05b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4188fa5b-0119-43dd-9c60-01dda2f9212d", "node_type": "1", "metadata": {}, "hash": "8babeb15154b929a74c648ac374258ccbd38ffb97d59a18506a171204f37c0a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12960, "end_char_idx": 16191, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We also recommend adding the `sampling_rate` argument in the feature extractor in order to better debug any silent errors that may occur.

```py
>>> audio_input = [dataset[0]["audio"]["array"]]
>>> feature_extractor(audio_input, sampling_rate=16000)
{'input_values': [array([ 3.8106556e-04,  2.7506407e-03,  2.8015103e-03, ...,
        5.6335266e-04,  4.6588284e-06, -1.7142107e-04], dtype=float32)]}
```

Just like the tokenizer, you can apply padding or truncation to handle variable sequences in a batch. Take a look at the sequence length of these two audio samples:

```py
>>> dataset[0]["audio"]["array"].shape
(173398,)

>>> dataset[1]["audio"]["array"].shape
(106496,)
```

Create a function to preprocess the dataset so the audio samples are the same lengths. Specify a maximum sample length, and the feature extractor will either pad or truncate the sequences to match it:

```py
>>> def preprocess_function(examples):
...     audio_arrays = [x["array"] for x in examples["audio"]]
...     inputs = feature_extractor(
...         audio_arrays,
...         sampling_rate=16000,
...         padding=True,
...         max_length=100000,
...         truncation=True,
...     )
...     return inputs
```

Apply the `preprocess_function` to the first few examples in the dataset:

```py
>>> processed_dataset = preprocess_function(dataset[:5])
```

The sample lengths are now the same and match the specified maximum length. You can pass your processed dataset to the model now!

```py
>>> processed_dataset["input_values"][0].shape
(100000,)

>>> processed_dataset["input_values"][1].shape
(100000,)
```

## Computer vision

For computer vision tasks, you'll need an [image processor](main_classes/image_processor) to prepare your dataset for the model.
Image preprocessing consists of several steps that convert images into the input expected by the model. These steps
include but are not limited to resizing, normalizing, color channel correction, and converting images to tensors.

<Tip>

Image preprocessing often follows some form of image augmentation. Both image preprocessing and image augmentation
transform image data, but they serve different purposes:

* Image augmentation alters images in a way that can help prevent overfitting and increase the robustness of the model. You can get creative in how you augment your data - adjust brightness and colors, crop, rotate, resize, zoom, etc. However, be mindful not to change the meaning of the images with your augmentations.
* Image preprocessing guarantees that the images match the model’s expected input format. When fine-tuning a computer vision model, images must be preprocessed exactly as when the model was initially trained.

You can use any library you like for image augmentation. For image preprocessing, use the `ImageProcessor` associated with the model.

</Tip>

Load the [food101](https://huggingface.co/datasets/food101) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use an image processor with computer vision datasets:

<Tip>

Use 🤗 Datasets `split` parameter to only load a small sample from the training split since the dataset is quite large!


key: _node_content
string_value:
{"id_": "4188fa5b-0119-43dd-9c60-01dda2f9212d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "851c446c-847e-4add-b90c-5d89e0e0a4bf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "77c133edd86254b4e6604a73f6651b639bf1bec5beae8a8d44ae1df6aedd9800", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fc1645b5-4b74-4ac0-bdbf-4116b85ab634", "node_type": "1", "metadata": {}, "hash": "1410408978c9a36806a714f31d670fdc4c6466b75f566cf9734810c5d6fd0b74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14389, "end_char_idx": 17918, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can pass your processed dataset to the model now!

```py
>>> processed_dataset["input_values"][0].shape
(100000,)

>>> processed_dataset["input_values"][1].shape
(100000,)
```

## Computer vision

For computer vision tasks, you'll need an [image processor](main_classes/image_processor) to prepare your dataset for the model.
Image preprocessing consists of several steps that convert images into the input expected by the model. These steps
include but are not limited to resizing, normalizing, color channel correction, and converting images to tensors.

<Tip>

Image preprocessing often follows some form of image augmentation. Both image preprocessing and image augmentation
transform image data, but they serve different purposes:

* Image augmentation alters images in a way that can help prevent overfitting and increase the robustness of the model. You can get creative in how you augment your data - adjust brightness and colors, crop, rotate, resize, zoom, etc. However, be mindful not to change the meaning of the images with your augmentations.
* Image preprocessing guarantees that the images match the model’s expected input format. When fine-tuning a computer vision model, images must be preprocessed exactly as when the model was initially trained.

You can use any library you like for image augmentation. For image preprocessing, use the `ImageProcessor` associated with the model.

</Tip>

Load the [food101](https://huggingface.co/datasets/food101) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use an image processor with computer vision datasets:

<Tip>

Use 🤗 Datasets `split` parameter to only load a small sample from the training split since the dataset is quite large!

</Tip>

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("food101", split="train[:100]")
```

Next, take a look at the image with 🤗 Datasets [`Image`](https://huggingface.co/docs/datasets/package_reference/main_classes?highlight=image#datasets.Image) feature:

```py
>>> dataset[0]["image"]
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vision-preprocess-tutorial.png"/>
</div>

Load the image processor with [`AutoImageProcessor.from_pretrained`]:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
```

First, let's add some image augmentation. You can use any library you prefer, but in this tutorial, we'll use torchvision's [`transforms`](https://pytorch.org/vision/stable/transforms.html) module. If you're interested in using another data augmentation library, learn how in the [Albumentations](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_albumentations.ipynb) or [Kornia notebooks](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_kornia.ipynb).

1. Here we use [`Compose`](https://pytorch.org/vision/master/generated/torchvision.transforms.Compose.html) to chain together a couple of
transforms - [`RandomResizedCrop`](https://pytorch.org/vision/main/generated/torchvision.transforms.RandomResizedCrop.html) and [`ColorJitter`](https://pytorch.org/vision/main/generated/torchvision.transforms.ColorJitter.html).
Note that for resizing, we can get the image size requirements from the `image_processor`.


key: _node_content
string_value:
{"id_": "fc1645b5-4b74-4ac0-bdbf-4116b85ab634", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4188fa5b-0119-43dd-9c60-01dda2f9212d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "3e67c67e9fbdc5404f93b4c31ac5eef19f2306d0de16e8650c0900d73ee6ed1b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fd306c5f-e8ff-4320-85f4-4514e4650fe2", "node_type": "1", "metadata": {}, "hash": "4ba95b02d45eb167035a2bb2d6ac07c2f79468a67297cc350a224c5b918ecd96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16193, "end_char_idx": 19543, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("food101", split="train[:100]")
```

Next, take a look at the image with 🤗 Datasets [`Image`](https://huggingface.co/docs/datasets/package_reference/main_classes?highlight=image#datasets.Image) feature:

```py
>>> dataset[0]["image"]
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vision-preprocess-tutorial.png"/>
</div>

Load the image processor with [`AutoImageProcessor.from_pretrained`]:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
```

First, let's add some image augmentation. You can use any library you prefer, but in this tutorial, we'll use torchvision's [`transforms`](https://pytorch.org/vision/stable/transforms.html) module. If you're interested in using another data augmentation library, learn how in the [Albumentations](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_albumentations.ipynb) or [Kornia notebooks](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_kornia.ipynb).

1. Here we use [`Compose`](https://pytorch.org/vision/master/generated/torchvision.transforms.Compose.html) to chain together a couple of
transforms - [`RandomResizedCrop`](https://pytorch.org/vision/main/generated/torchvision.transforms.RandomResizedCrop.html) and [`ColorJitter`](https://pytorch.org/vision/main/generated/torchvision.transforms.ColorJitter.html).
Note that for resizing, we can get the image size requirements from the `image_processor`. For some models, an exact height and
width are expected, for others only the `shortest_edge` is defined.

```py
>>> from torchvision.transforms import RandomResizedCrop, ColorJitter, Compose

>>> size = (
...     image_processor.size["shortest_edge"]
...     if "shortest_edge" in image_processor.size
...     else (image_processor.size["height"], image_processor.size["width"])
... )

>>> _transforms = Compose([RandomResizedCrop(size), ColorJitter(brightness=0.5, hue=0.5)])
```

2. The model accepts [`pixel_values`](model_doc/vision-encoder-decoder#transformers.VisionEncoderDecoderModel.forward.pixel_values)
as its input. `ImageProcessor` can take care of normalizing the images, and generating appropriate tensors.
Create a function that combines image augmentation and image preprocessing for a batch of images and generates `pixel_values`:

```py
>>> def transforms(examples):
...     images = [_transforms(img.convert("RGB")) for img in examples["image"]]
...     examples["pixel_values"] = image_processor(images, do_resize=False, return_tensors="pt")["pixel_values"]
...     return examples
```

<Tip>

In the example above we set `do_resize=False` because we have already resized the images in the image augmentation transformation,
and leveraged the `size` attribute from the appropriate `image_processor`. If you do not resize images during image augmentation,
leave this parameter out. By default, `ImageProcessor` will handle the resizing.

If you wish to normalize images as a part of the augmentation transformation, use the `image_processor.image_mean`,
and `image_processor.image_std` values.
</Tip>

3.


key: _node_content
string_value:
{"id_": "fd306c5f-e8ff-4320-85f4-4514e4650fe2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fc1645b5-4b74-4ac0-bdbf-4116b85ab634", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "bbc6e5cc6f03d73e736e09800cd410f2ce36b9467f71570aeda11642bb0a7138", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7e18b06-79bc-48e3-a80e-4c9825669bcd", "node_type": "1", "metadata": {}, "hash": "dd0abd2823e1c334ee40889278d1ca109fe21917312dd7f23cf17b401dd8e964", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17828, "end_char_idx": 20921, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that for resizing, we can get the image size requirements from the `image_processor`. For some models, an exact height and
width are expected, for others only the `shortest_edge` is defined.

```py
>>> from torchvision.transforms import RandomResizedCrop, ColorJitter, Compose

>>> size = (
...     image_processor.size["shortest_edge"]
...     if "shortest_edge" in image_processor.size
...     else (image_processor.size["height"], image_processor.size["width"])
... )

>>> _transforms = Compose([RandomResizedCrop(size), ColorJitter(brightness=0.5, hue=0.5)])
```

2. The model accepts [`pixel_values`](model_doc/vision-encoder-decoder#transformers.VisionEncoderDecoderModel.forward.pixel_values)
as its input. `ImageProcessor` can take care of normalizing the images, and generating appropriate tensors.
Create a function that combines image augmentation and image preprocessing for a batch of images and generates `pixel_values`:

```py
>>> def transforms(examples):
...     images = [_transforms(img.convert("RGB")) for img in examples["image"]]
...     examples["pixel_values"] = image_processor(images, do_resize=False, return_tensors="pt")["pixel_values"]
...     return examples
```

<Tip>

In the example above we set `do_resize=False` because we have already resized the images in the image augmentation transformation,
and leveraged the `size` attribute from the appropriate `image_processor`. If you do not resize images during image augmentation,
leave this parameter out. By default, `ImageProcessor` will handle the resizing.

If you wish to normalize images as a part of the augmentation transformation, use the `image_processor.image_mean`,
and `image_processor.image_std` values.
</Tip>

3. Then use 🤗 Datasets[`~datasets.Dataset.set_transform`] to apply the transforms on the fly:
```py
>>> dataset.set_transform(transforms)
```

4. Now when you access the image, you'll notice the image processor has added `pixel_values`. You can pass your processed dataset to the model now!

```py
>>> dataset[0].keys()
```

Here is what the image looks like after the transforms are applied. The image has been randomly cropped and it's color properties are different.

```py
>>> import numpy as np
>>> import matplotlib.pyplot as plt

>>> img = dataset[0]["pixel_values"]
>>> plt.imshow(img.permute(1, 2, 0))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/preprocessed_image.png"/>
</div>

<Tip>

For tasks like object detection, semantic segmentation, instance segmentation, and panoptic segmentation, `ImageProcessor`
offers post processing methods. These methods convert model's raw outputs into meaningful predictions such as bounding boxes,
or segmentation maps.

</Tip>

### Pad

In some cases, for instance, when fine-tuning [DETR](./model_doc/detr), the model applies scale augmentation at training
time. This may cause images to be different sizes in a batch. You can use [`DetrImageProcessor.pad`]
from [`DetrImageProcessor`] and define a custom `collate_fn` to batch images together.


key: _node_content
string_value:
{"id_": "f7e18b06-79bc-48e3-a80e-4c9825669bcd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fd306c5f-e8ff-4320-85f4-4514e4650fe2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "77dde90ed4160e8c6d60b85c74af9a99898accf63c156343bb79f4fc92bc0584", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fc6a0b42-df99-446d-b8e8-85f4eb79cab6", "node_type": "1", "metadata": {}, "hash": "c39f41d58c94f8d1a15fe5885021a22add22dfc84ed9e62378aff7d4f79c0e70", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19240, "end_char_idx": 21570, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you do not resize images during image augmentation,
leave this parameter out. By default, `ImageProcessor` will handle the resizing.

If you wish to normalize images as a part of the augmentation transformation, use the `image_processor.image_mean`,
and `image_processor.image_std` values.
</Tip>

3. Then use 🤗 Datasets[`~datasets.Dataset.set_transform`] to apply the transforms on the fly:
```py
>>> dataset.set_transform(transforms)
```

4. Now when you access the image, you'll notice the image processor has added `pixel_values`. You can pass your processed dataset to the model now!

```py
>>> dataset[0].keys()
```

Here is what the image looks like after the transforms are applied. The image has been randomly cropped and it's color properties are different.

```py
>>> import numpy as np
>>> import matplotlib.pyplot as plt

>>> img = dataset[0]["pixel_values"]
>>> plt.imshow(img.permute(1, 2, 0))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/preprocessed_image.png"/>
</div>

<Tip>

For tasks like object detection, semantic segmentation, instance segmentation, and panoptic segmentation, `ImageProcessor`
offers post processing methods. These methods convert model's raw outputs into meaningful predictions such as bounding boxes,
or segmentation maps.

</Tip>

### Pad

In some cases, for instance, when fine-tuning [DETR](./model_doc/detr), the model applies scale augmentation at training
time. This may cause images to be different sizes in a batch. You can use [`DetrImageProcessor.pad`]
from [`DetrImageProcessor`] and define a custom `collate_fn` to batch images together.

```py
>>> def collate_fn(batch):
...     pixel_values = [item["pixel_values"] for item in batch]
...     encoding = image_processor.pad(pixel_values, return_tensors="pt")
...     labels = [item["labels"] for item in batch]
...     batch = {}
...     batch["pixel_values"] = encoding["pixel_values"]
...     batch["pixel_mask"] = encoding["pixel_mask"]
...     batch["labels"] = labels
...     return batch
```

## Multimodal

For tasks involving multimodal inputs, you'll need a [processor](main_classes/processors) to prepare your dataset for the model. A processor couples together two processing objects such as tokenizer and feature extractor.


key: _node_content
string_value:
{"id_": "fc6a0b42-df99-446d-b8e8-85f4eb79cab6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7e18b06-79bc-48e3-a80e-4c9825669bcd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "0063149ed445dc0c76f115253cd977d064d5fc2df24b2160ce89d9c107628802", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b3afd71d-5a31-4f71-80a1-372f14adacdd", "node_type": "1", "metadata": {}, "hash": "b7196abb711ab229bf55d28e6fc41efa5416af045ca48ec302caa817d181487d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19833, "end_char_idx": 22892, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> dataset[0].keys()
```

Here is what the image looks like after the transforms are applied. The image has been randomly cropped and it's color properties are different.

```py
>>> import numpy as np
>>> import matplotlib.pyplot as plt

>>> img = dataset[0]["pixel_values"]
>>> plt.imshow(img.permute(1, 2, 0))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/preprocessed_image.png"/>
</div>

<Tip>

For tasks like object detection, semantic segmentation, instance segmentation, and panoptic segmentation, `ImageProcessor`
offers post processing methods. These methods convert model's raw outputs into meaningful predictions such as bounding boxes,
or segmentation maps.

</Tip>

### Pad

In some cases, for instance, when fine-tuning [DETR](./model_doc/detr), the model applies scale augmentation at training
time. This may cause images to be different sizes in a batch. You can use [`DetrImageProcessor.pad`]
from [`DetrImageProcessor`] and define a custom `collate_fn` to batch images together.

```py
>>> def collate_fn(batch):
...     pixel_values = [item["pixel_values"] for item in batch]
...     encoding = image_processor.pad(pixel_values, return_tensors="pt")
...     labels = [item["labels"] for item in batch]
...     batch = {}
...     batch["pixel_values"] = encoding["pixel_values"]
...     batch["pixel_mask"] = encoding["pixel_mask"]
...     batch["labels"] = labels
...     return batch
```

## Multimodal

For tasks involving multimodal inputs, you'll need a [processor](main_classes/processors) to prepare your dataset for the model. A processor couples together two processing objects such as tokenizer and feature extractor.

Load the [LJ Speech](https://huggingface.co/datasets/lj_speech) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use a processor for automatic speech recognition (ASR):

```py
>>> from datasets import load_dataset

>>> lj_speech = load_dataset("lj_speech", split="train")
```

For ASR, you're mainly focused on `audio` and `text` so you can remove the other columns:

```py
>>> lj_speech = lj_speech.map(remove_columns=["file", "id", "normalized_text"])
```

Now take a look at the `audio` and `text` columns:

```py
>>> lj_speech[0]["audio"]
{'array': array([-7.3242188e-04, -7.6293945e-04, -6.4086914e-04, ...,
         7.3242188e-04,  2.1362305e-04,  6.1035156e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/917ece08c95cf0c4115e45294e3cd0dee724a1165b7fc11798369308a465bd26/LJSpeech-1.1/wavs/LJ001-0001.wav',
 'sampling_rate': 22050}

>>> lj_speech[0]["text"]
'Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition'
```

Remember you should always [resample](preprocessing#audio) your audio dataset's sampling rate to match the sampling rate of the dataset used to pretrain a model!


key: _node_content
string_value:
{"id_": "b3afd71d-5a31-4f71-80a1-372f14adacdd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c77c70d4-491e-4d74-ae17-8f1df6277010", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fc6a0b42-df99-446d-b8e8-85f4eb79cab6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "e286f602d198cd0c06b5064766d12afca550545a8737abdf2be565c382f6eb20", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21572, "end_char_idx": 23806, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Load the [LJ Speech](https://huggingface.co/datasets/lj_speech) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use a processor for automatic speech recognition (ASR):

```py
>>> from datasets import load_dataset

>>> lj_speech = load_dataset("lj_speech", split="train")
```

For ASR, you're mainly focused on `audio` and `text` so you can remove the other columns:

```py
>>> lj_speech = lj_speech.map(remove_columns=["file", "id", "normalized_text"])
```

Now take a look at the `audio` and `text` columns:

```py
>>> lj_speech[0]["audio"]
{'array': array([-7.3242188e-04, -7.6293945e-04, -6.4086914e-04, ...,
         7.3242188e-04,  2.1362305e-04,  6.1035156e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/917ece08c95cf0c4115e45294e3cd0dee724a1165b7fc11798369308a465bd26/LJSpeech-1.1/wavs/LJ001-0001.wav',
 'sampling_rate': 22050}

>>> lj_speech[0]["text"]
'Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition'
```

Remember you should always [resample](preprocessing#audio) your audio dataset's sampling rate to match the sampling rate of the dataset used to pretrain a model!

```py
>>> lj_speech = lj_speech.cast_column("audio", Audio(sampling_rate=16_000))
```

Load a processor with [`AutoProcessor.from_pretrained`]:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base-960h")
```

1. Create a function to process the audio data contained in `array` to `input_values`, and tokenize `text` to `labels`. These are the inputs to the model:

```py
>>> def prepare_dataset(example):
...     audio = example["audio"]

...     example.update(processor(audio=audio["array"], text=example["text"], sampling_rate=16000))

...     return example
```

2. Apply the `prepare_dataset` function to a sample:

```py
>>> prepare_dataset(lj_speech[0])
```

The processor has now added `input_values` and `labels`, and the sampling rate has also been correctly downsampled to 16kHz. You can pass your processed dataset to the model now!


key: _node_content
string_value:
{"id_": "c3dedbf7-9b19-4664-b540-e5e5406b0e44", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/sagemaker", "title": "Run training on Amazon SageMaker", "tokens": 278, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9d7a918b-1ebb-439d-abd1-26f7b183ffa5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/sagemaker", "title": "Run training on Amazon SageMaker", "tokens": 278, "retrieve_doc": true, "source": "transformers"}, "hash": "7698b2ff621f3e0a64efa530a64dc41f68dc6ea4f228f13ba9b6e5f358029a6b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1220, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!---
Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Run training on Amazon SageMaker

The documentation has been moved to [hf.co/docs/sagemaker](https://huggingface.co/docs/sagemaker). This page will be removed in `transformers` 5.0. 

### Table of Content

- [Train Hugging Face models on Amazon SageMaker with the SageMaker Python SDK](https://huggingface.co/docs/sagemaker/train)
- [Deploy Hugging Face models to Amazon SageMaker with the SageMaker Python SDK](https://huggingface.co/docs/sagemaker/inference)


key: _node_content
string_value:
{"id_": "90ae211d-08ca-4619-90f4-9ef328c55280", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "95e4d21b-5430-4814-a1a7-a0751960cd7f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "hash": "60d389b98b70973e18401b03836e98d77a216158a1e16fe011c2142e03543ba8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8790ede3-9801-416a-a28b-d7d67d3a4da0", "node_type": "1", "metadata": {}, "hash": "b6923e4934a3ea39b940ba629f42969d86c162b763559003e2725dbe09ee9fba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3390, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Share a model

The last two tutorials showed how you can fine-tune a model with PyTorch, Keras, and 🤗 Accelerate for distributed setups. The next step is to share your model with the community! At Hugging Face, we believe in openly sharing knowledge and resources to democratize artificial intelligence for everyone. We encourage you to consider sharing your model with the community to help others save time and resources.

In this tutorial, you will learn two methods for sharing a trained or fine-tuned model on the [Model Hub](https://huggingface.co/models):

- Programmatically push your files to the Hub.
- Drag-and-drop your files to the Hub with the web interface.

<iframe width="560" height="315" src="https://www.youtube.com/embed/XvSGPZFEjDY" title="YouTube video player"
frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope;
picture-in-picture" allowfullscreen></iframe>

<Tip>

To share a model with the community, you need an account on [huggingface.co](https://huggingface.co/join). You can also join an existing organization or create a new one.

</Tip>

## Repository features

Each repository on the Model Hub behaves like a typical GitHub repository. Our repositories offer versioning, commit history, and the ability to visualize differences.

The Model Hub's built-in versioning is based on git and [git-lfs](https://git-lfs.github.com/). In other words, you can treat one model as one repository, enabling greater access control and scalability. Version control allows *revisions*, a method for pinning a specific version of a model with a commit hash, tag or branch.

As a result, you can load a specific model version with the `revision` parameter:

```py
>>> model = AutoModel.from_pretrained(
...     "julien-c/EsperBERTo-small", revision="v2.0.1"  # tag name, or branch name, or commit hash
... )
```

Files are also easily edited in a repository, and you can view the commit history as well as the difference:

![vis_diff](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vis_diff.png)

## Setup

Before sharing a model to the Hub, you will need your Hugging Face credentials. If you have access to a terminal, run the following command in the virtual environment where 🤗 Transformers is installed. This will store your access token in your Hugging Face cache folder (`~/.cache/` by default):

```bash
huggingface-cli login
```

If you are using a notebook like Jupyter or Colaboratory, make sure you have the [`huggingface_hub`](https://huggingface.co/docs/hub/adding-a-library) library installed. This library allows you to programmatically interact with the Hub.

```bash
pip install huggingface_hub
```

Then use `notebook_login` to sign-in to the Hub, and follow the link [here](https://huggingface.co/settings/token) to generate a token to login with:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Convert a model for all frameworks

To ensure your model can be used by someone working with a different framework, we recommend you convert and upload your model with both PyTorch and TensorFlow checkpoints. While users are still able to load your model from a different framework if you skip this step, it will be slower because 🤗 Transformers will need to convert the checkpoint on-the-fly.

Converting a checkpoint for another framework is easy.


key: _node_content
string_value:
{"id_": "8790ede3-9801-416a-a28b-d7d67d3a4da0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "95e4d21b-5430-4814-a1a7-a0751960cd7f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "hash": "60d389b98b70973e18401b03836e98d77a216158a1e16fe011c2142e03543ba8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90ae211d-08ca-4619-90f4-9ef328c55280", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "hash": "57b8c29332d2ed8f5a0c549207b81bb957198dc058215c62bad3e579996eb51d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8381d8d6-a7a2-41ca-99ef-b93e17652b9a", "node_type": "1", "metadata": {}, "hash": "9a8fda1c3a5dfa0893a2d863d0e8026ee225667854bf81254c53b2af6b804e33", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1982, "end_char_idx": 4908, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[vis_diff](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vis_diff.png)

## Setup

Before sharing a model to the Hub, you will need your Hugging Face credentials. If you have access to a terminal, run the following command in the virtual environment where 🤗 Transformers is installed. This will store your access token in your Hugging Face cache folder (`~/.cache/` by default):

```bash
huggingface-cli login
```

If you are using a notebook like Jupyter or Colaboratory, make sure you have the [`huggingface_hub`](https://huggingface.co/docs/hub/adding-a-library) library installed. This library allows you to programmatically interact with the Hub.

```bash
pip install huggingface_hub
```

Then use `notebook_login` to sign-in to the Hub, and follow the link [here](https://huggingface.co/settings/token) to generate a token to login with:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Convert a model for all frameworks

To ensure your model can be used by someone working with a different framework, we recommend you convert and upload your model with both PyTorch and TensorFlow checkpoints. While users are still able to load your model from a different framework if you skip this step, it will be slower because 🤗 Transformers will need to convert the checkpoint on-the-fly.

Converting a checkpoint for another framework is easy. Make sure you have PyTorch and TensorFlow installed (see [here](installation) for installation instructions), and then find the specific model for your task in the other framework.

<frameworkcontent>
<pt>
Specify `from_tf=True` to convert a checkpoint from TensorFlow to PyTorch:

```py
>>> pt_model = DistilBertForSequenceClassification.from_pretrained("path/to/awesome-name-you-picked", from_tf=True)
>>> pt_model.save_pretrained("path/to/awesome-name-you-picked")
```
</pt>
<tf>
Specify `from_pt=True` to convert a checkpoint from PyTorch to TensorFlow:

```py
>>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("path/to/awesome-name-you-picked", from_pt=True)
```

Then you can save your new TensorFlow model with its new checkpoint:

```py
>>> tf_model.save_pretrained("path/to/awesome-name-you-picked")
```
</tf>
<jax>
If a model is available in Flax, you can also convert a checkpoint from PyTorch to Flax:

```py
>>> flax_model = FlaxDistilBertForSequenceClassification.from_pretrained(
...     "path/to/awesome-name-you-picked", from_pt=True
... )
```
</jax>
</frameworkcontent>

## Push a model during training

<frameworkcontent>
<pt>
<Youtube id="Z1-XMy-GNLQ"/>

Sharing a model to the Hub is as simple as adding an extra parameter or callback. Remember from the [fine-tuning tutorial](training), the [`TrainingArguments`] class is where you specify hyperparameters and additional training options. One of these training options includes the ability to push a model directly to the Hub.


key: _node_content
string_value:
{"id_": "8381d8d6-a7a2-41ca-99ef-b93e17652b9a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "95e4d21b-5430-4814-a1a7-a0751960cd7f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "hash": "60d389b98b70973e18401b03836e98d77a216158a1e16fe011c2142e03543ba8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8790ede3-9801-416a-a28b-d7d67d3a4da0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "hash": "4f2a8c6a6035561998831b678f85821e02e3e30f797b1681d263e47da81e4a78", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d1df8d3e-9b87-47cd-ab21-1fe00a91e103", "node_type": "1", "metadata": {}, "hash": "359803c6316cafb665e7d0c7efae85dc176287329b68d5a91cf8411c971d0036", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3336, "end_char_idx": 6479, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Converting a checkpoint for another framework is easy. Make sure you have PyTorch and TensorFlow installed (see [here](installation) for installation instructions), and then find the specific model for your task in the other framework.

<frameworkcontent>
<pt>
Specify `from_tf=True` to convert a checkpoint from TensorFlow to PyTorch:

```py
>>> pt_model = DistilBertForSequenceClassification.from_pretrained("path/to/awesome-name-you-picked", from_tf=True)
>>> pt_model.save_pretrained("path/to/awesome-name-you-picked")
```
</pt>
<tf>
Specify `from_pt=True` to convert a checkpoint from PyTorch to TensorFlow:

```py
>>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("path/to/awesome-name-you-picked", from_pt=True)
```

Then you can save your new TensorFlow model with its new checkpoint:

```py
>>> tf_model.save_pretrained("path/to/awesome-name-you-picked")
```
</tf>
<jax>
If a model is available in Flax, you can also convert a checkpoint from PyTorch to Flax:

```py
>>> flax_model = FlaxDistilBertForSequenceClassification.from_pretrained(
...     "path/to/awesome-name-you-picked", from_pt=True
... )
```
</jax>
</frameworkcontent>

## Push a model during training

<frameworkcontent>
<pt>
<Youtube id="Z1-XMy-GNLQ"/>

Sharing a model to the Hub is as simple as adding an extra parameter or callback. Remember from the [fine-tuning tutorial](training), the [`TrainingArguments`] class is where you specify hyperparameters and additional training options. One of these training options includes the ability to push a model directly to the Hub. Set `push_to_hub=True` in your [`TrainingArguments`]:

```py
>>> training_args = TrainingArguments(output_dir="my-awesome-model", push_to_hub=True)
```

Pass your training arguments as usual to [`Trainer`]:

```py
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... )
```

After you fine-tune your model, call [`~transformers.Trainer.push_to_hub`] on [`Trainer`] to push the trained model to the Hub. 🤗 Transformers will even automatically add training hyperparameters, training results and framework versions to your model card!

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
Share a model to the Hub with [`PushToHubCallback`]. In the [`PushToHubCallback`] function, add:

- An output directory for your model.
- A tokenizer.
- The `hub_model_id`, which is your Hub username and model name.

```py
>>> from transformers import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="./your_model_save_path", tokenizer=tokenizer, hub_model_id="your-username/my-awesome-model"
... )
```

Add the callback to [`fit`](https://keras.io/api/models/model_training_apis/), and 🤗 Transformers will push the trained model to the Hub:

```py
>>> model.fit(tf_train_dataset, validation_data=tf_validation_dataset, epochs=3, callbacks=push_to_hub_callback)
```
</tf>
</frameworkcontent>

## Use the `push_to_hub` function

You can also call `push_to_hub` directly on your model to upload it to the Hub.


key: _node_content
string_value:
{"id_": "d1df8d3e-9b87-47cd-ab21-1fe00a91e103", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "95e4d21b-5430-4814-a1a7-a0751960cd7f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "hash": "60d389b98b70973e18401b03836e98d77a216158a1e16fe011c2142e03543ba8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8381d8d6-a7a2-41ca-99ef-b93e17652b9a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "hash": "1e1c0e14cb782f47f87f77a50ff9ee27972665a14048909187b1b9f1446f512b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2b462324-57e6-4136-b2f2-a10c3d534fc6", "node_type": "1", "metadata": {}, "hash": "e79843da10f7db55e0b89d3b645885e719cdc4703c2e0c9759e2c7a14ae32657", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4821, "end_char_idx": 7965, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
One of these training options includes the ability to push a model directly to the Hub. Set `push_to_hub=True` in your [`TrainingArguments`]:

```py
>>> training_args = TrainingArguments(output_dir="my-awesome-model", push_to_hub=True)
```

Pass your training arguments as usual to [`Trainer`]:

```py
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... )
```

After you fine-tune your model, call [`~transformers.Trainer.push_to_hub`] on [`Trainer`] to push the trained model to the Hub. 🤗 Transformers will even automatically add training hyperparameters, training results and framework versions to your model card!

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
Share a model to the Hub with [`PushToHubCallback`]. In the [`PushToHubCallback`] function, add:

- An output directory for your model.
- A tokenizer.
- The `hub_model_id`, which is your Hub username and model name.

```py
>>> from transformers import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="./your_model_save_path", tokenizer=tokenizer, hub_model_id="your-username/my-awesome-model"
... )
```

Add the callback to [`fit`](https://keras.io/api/models/model_training_apis/), and 🤗 Transformers will push the trained model to the Hub:

```py
>>> model.fit(tf_train_dataset, validation_data=tf_validation_dataset, epochs=3, callbacks=push_to_hub_callback)
```
</tf>
</frameworkcontent>

## Use the `push_to_hub` function

You can also call `push_to_hub` directly on your model to upload it to the Hub.

Specify your model name in `push_to_hub`:

```py
>>> pt_model.push_to_hub("my-awesome-model")
```

This creates a repository under your username with the model name `my-awesome-model`. Users can now load your model with the `from_pretrained` function:

```py
>>> from transformers import AutoModel

>>> model = AutoModel.from_pretrained("your_username/my-awesome-model")
```

If you belong to an organization and want to push your model under the organization name instead, just add it to the `repo_id`:

```py
>>> pt_model.push_to_hub("my-awesome-org/my-awesome-model")
```

The `push_to_hub` function can also be used to add other files to a model repository. For example, add a tokenizer to a model repository:

```py
>>> tokenizer.push_to_hub("my-awesome-model")
```

Or perhaps you'd like to add the TensorFlow version of your fine-tuned PyTorch model:

```py
>>> tf_model.push_to_hub("my-awesome-model")
```

Now when you navigate to your Hugging Face profile, you should see your newly created model repository. Clicking on the **Files** tab will display all the files you've uploaded to the repository.

For more details on how to create and upload files to a repository, refer to the Hub documentation [here](https://huggingface.co/docs/hub/how-to-upstream).

## Upload with the web interface

Users who prefer a no-code approach are able to upload a model through the Hub's web interface. Visit [huggingface.co/new](https://huggingface.co/new) to create a new repository:

!


key: _node_content
string_value:
{"id_": "2b462324-57e6-4136-b2f2-a10c3d534fc6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "95e4d21b-5430-4814-a1a7-a0751960cd7f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "hash": "60d389b98b70973e18401b03836e98d77a216158a1e16fe011c2142e03543ba8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d1df8d3e-9b87-47cd-ab21-1fe00a91e103", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2341, "retrieve_doc": true, "source": "transformers"}, "hash": "ed050444d4856a27ac747df271231c87d2eb738171580b84a5bd1d2a99cd3322", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6481, "end_char_idx": 9462, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Specify your model name in `push_to_hub`:

```py
>>> pt_model.push_to_hub("my-awesome-model")
```

This creates a repository under your username with the model name `my-awesome-model`. Users can now load your model with the `from_pretrained` function:

```py
>>> from transformers import AutoModel

>>> model = AutoModel.from_pretrained("your_username/my-awesome-model")
```

If you belong to an organization and want to push your model under the organization name instead, just add it to the `repo_id`:

```py
>>> pt_model.push_to_hub("my-awesome-org/my-awesome-model")
```

The `push_to_hub` function can also be used to add other files to a model repository. For example, add a tokenizer to a model repository:

```py
>>> tokenizer.push_to_hub("my-awesome-model")
```

Or perhaps you'd like to add the TensorFlow version of your fine-tuned PyTorch model:

```py
>>> tf_model.push_to_hub("my-awesome-model")
```

Now when you navigate to your Hugging Face profile, you should see your newly created model repository. Clicking on the **Files** tab will display all the files you've uploaded to the repository.

For more details on how to create and upload files to a repository, refer to the Hub documentation [here](https://huggingface.co/docs/hub/how-to-upstream).

## Upload with the web interface

Users who prefer a no-code approach are able to upload a model through the Hub's web interface. Visit [huggingface.co/new](https://huggingface.co/new) to create a new repository:

![new_model_repo](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/new_model_repo.png)

From here, add some information about your model:

- Select the **owner** of the repository. This can be yourself or any of the organizations you belong to.
- Pick a name for your model, which will also be the repository name.
- Choose whether your model is public or private.
- Specify the license usage for your model.

Now click on the **Files** tab and click on the **Add file** button to upload a new file to your repository. Then drag-and-drop a file to upload and add a commit message.

![upload_file](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upload_file.png)

## Add a model card

To make sure users understand your model's capabilities, limitations, potential biases and ethical considerations, please add a model card to your repository. The model card is defined in the `README.md` file. You can add a model card by:

* Manually creating and uploading a `README.md` file.
* Clicking on the **Edit model card** button in your model repository.

Take a look at the DistilBert [model card](https://huggingface.co/distilbert/distilbert-base-uncased) for a good example of the type of information a model card should include. For more details about other options you can control in the `README.md` file such as a model's carbon footprint or widget examples, refer to the documentation [here](https://huggingface.co/docs/hub/models-cards).


key: _node_content
string_value:
{"id_": "9cbf597b-562c-47df-97c6-33d0bdbd9b00", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "807c3bb3-e0b5-40ca-8c6b-f78805efdf85", "node_type": "1", "metadata": {}, "hash": "9acc167e50e9a635700da48f9c51c2e752c556256a126da90553b5316b2be87b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2914, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPU inference

GPUs are the standard choice of hardware for machine learning, unlike CPUs, because they are optimized for memory bandwidth and parallelism. To keep up with the larger sizes of modern models or to run these large models on existing and older hardware, there are several optimizations you can use to speed up GPU inference. In this guide, you'll learn how to use FlashAttention-2 (a more memory-efficient attention mechanism), BetterTransformer (a PyTorch native fastpath execution), and bitsandbytes to quantize your model to a lower precision. Finally, learn how to use 🤗 Optimum to accelerate inference with ONNX Runtime on Nvidia and AMD GPUs.

<Tip>

The majority of the optimizations described here also apply to multi-GPU setups!

</Tip>

## FlashAttention-2

<Tip>

FlashAttention-2 is experimental and may change considerably in future versions.

</Tip>

[FlashAttention-2](https://huggingface.co/papers/2205.14135) is a faster and more efficient implementation of the standard attention mechanism that can significantly speedup inference by:

1. additionally parallelizing the attention computation over sequence length
2. partitioning the work between GPU threads to reduce communication and shared memory reads/writes between them

FlashAttention-2 is currently supported for the following architectures:
* [Bark](https://huggingface.co/docs/transformers/model_doc/bark#transformers.BarkModel)
* [Bart](https://huggingface.co/docs/transformers/model_doc/bart#transformers.BartModel)
* [Chameleon](https://huggingface.co/docs/transformers/model_doc/chameleon#transformers.Chameleon)
* [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPModel)
* [Cohere](https://huggingface.co/docs/transformers/model_doc/cohere#transformers.CohereModel)
* [Dbrx](https://huggingface.co/docs/transformers/model_doc/dbrx#transformers.DbrxModel)
* [DistilBert](https://huggingface.co/docs/transformers/model_doc/distilbert#transformers.DistilBertModel)
* [Gemma](https://huggingface.co/docs/transformers/model_doc/gemma#transformers.GemmaModel)
* [Gemma2](https://huggingface.co/docs/transformers/model_doc/gemma2#transformers.Gemma2Model)
* [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)
* [GPTBigCode](https://huggingface.co/docs/transformers/model_doc/gpt_bigcode#transformers.GPTBigCodeModel)
* [GPTNeo](https://huggingface.co/docs/transformers/model_doc/gpt_neo#transformers.GPTNeoModel)
* [GPTNeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox#transformers.GPTNeoXModel)
* [GPT-J](https://huggingface.co/docs/transformers/model_doc/gptj#transformers.GPTJModel)
* [Idefics2](https://huggingface.co/docs/transformers/model_doc/idefics2#transformers.Idefics2Model)
* [Falcon](https://huggingface.co/docs/transformers/model_doc/falcon#transformers.FalconModel)
* [JetMoe](https://huggingface.co/docs/transformers/model_doc/jetmoe#transformers.


key: _node_content
string_value:
{"id_": "807c3bb3-e0b5-40ca-8c6b-f78805efdf85", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9cbf597b-562c-47df-97c6-33d0bdbd9b00", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "16815aefd8397ab4e49fd404de266f531d973053bcfd90dc2a1a9f208b019a60", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c2df8d2a-7028-4ad3-9379-43e0506774a2", "node_type": "1", "metadata": {}, "hash": "5284c40f91c497f5a1f21011dc8fcca97ac1bbf4c1ea7b25d1c270d2273c8b5b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1730, "end_char_idx": 4114, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/cohere#transformers.CohereModel)
* [Dbrx](https://huggingface.co/docs/transformers/model_doc/dbrx#transformers.DbrxModel)
* [DistilBert](https://huggingface.co/docs/transformers/model_doc/distilbert#transformers.DistilBertModel)
* [Gemma](https://huggingface.co/docs/transformers/model_doc/gemma#transformers.GemmaModel)
* [Gemma2](https://huggingface.co/docs/transformers/model_doc/gemma2#transformers.Gemma2Model)
* [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)
* [GPTBigCode](https://huggingface.co/docs/transformers/model_doc/gpt_bigcode#transformers.GPTBigCodeModel)
* [GPTNeo](https://huggingface.co/docs/transformers/model_doc/gpt_neo#transformers.GPTNeoModel)
* [GPTNeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox#transformers.GPTNeoXModel)
* [GPT-J](https://huggingface.co/docs/transformers/model_doc/gptj#transformers.GPTJModel)
* [Idefics2](https://huggingface.co/docs/transformers/model_doc/idefics2#transformers.Idefics2Model)
* [Falcon](https://huggingface.co/docs/transformers/model_doc/falcon#transformers.FalconModel)
* [JetMoe](https://huggingface.co/docs/transformers/model_doc/jetmoe#transformers.JetMoeModel)
* [Jamba](https://huggingface.co/docs/transformers/model_doc/jamba#transformers.JambaModel)
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [Llava](https://huggingface.co/docs/transformers/model_doc/llava)
* [Llava-NeXT](https://huggingface.co/docs/transformers/model_doc/llava_next)
* [Llava-NeXT-Video](https://huggingface.co/docs/transformers/model_doc/llava_next_video)
* [VipLlava](https://huggingface.co/docs/transformers/model_doc/vipllava)
* [VideoLlava](https://huggingface.co/docs/transformers/model_doc/video_llava)
* [M2M100](https://huggingface.co/docs/transformers/model_doc/m2m_100)
* [MBart](https://huggingface.co/docs/transformers/model_doc/mbart#transformers.MBartModel)
* [Mistral](https://huggingface.co/docs/transformers/model_doc/mistral#transformers.MistralModel)
* [Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral#transformers.MixtralModel)
* [Musicgen](https://huggingface.co/docs/transformers/model_doc/musicgen#transformers.MusicgenModel)
* [MusicGen Melody](https://huggingface.co/docs/transformers/model_doc/musicgen_melody#transformers.MusicgenMelodyModel)
* [Nemotron](https://huggingface.


key: _node_content
string_value:
{"id_": "c2df8d2a-7028-4ad3-9379-43e0506774a2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "807c3bb3-e0b5-40ca-8c6b-f78805efdf85", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "a2d3ac4ed3e51eeda70a2be1a21fc2e94d0692779cd8e66e6cd66bbbdfcf5dc0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa750ee1-c4ff-4197-8fc8-5e0d4552826e", "node_type": "1", "metadata": {}, "hash": "e12c2e5ae5eec2351cb99dc0d6744aecff323f08617325e5ca05977a79d06aab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2863, "end_char_idx": 5284, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/jetmoe#transformers.JetMoeModel)
* [Jamba](https://huggingface.co/docs/transformers/model_doc/jamba#transformers.JambaModel)
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [Llava](https://huggingface.co/docs/transformers/model_doc/llava)
* [Llava-NeXT](https://huggingface.co/docs/transformers/model_doc/llava_next)
* [Llava-NeXT-Video](https://huggingface.co/docs/transformers/model_doc/llava_next_video)
* [VipLlava](https://huggingface.co/docs/transformers/model_doc/vipllava)
* [VideoLlava](https://huggingface.co/docs/transformers/model_doc/video_llava)
* [M2M100](https://huggingface.co/docs/transformers/model_doc/m2m_100)
* [MBart](https://huggingface.co/docs/transformers/model_doc/mbart#transformers.MBartModel)
* [Mistral](https://huggingface.co/docs/transformers/model_doc/mistral#transformers.MistralModel)
* [Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral#transformers.MixtralModel)
* [Musicgen](https://huggingface.co/docs/transformers/model_doc/musicgen#transformers.MusicgenModel)
* [MusicGen Melody](https://huggingface.co/docs/transformers/model_doc/musicgen_melody#transformers.MusicgenMelodyModel)
* [Nemotron](https://huggingface.co/docs/transformers/model_doc/nemotron)
* [NLLB](https://huggingface.co/docs/transformers/model_doc/nllb)
* [OLMo](https://huggingface.co/docs/transformers/model_doc/olmo#transformers.OlmoModel)
* [OPT](https://huggingface.co/docs/transformers/model_doc/opt#transformers.OPTModel)
* [Phi](https://huggingface.co/docs/transformers/model_doc/phi#transformers.PhiModel)
* [Phi3](https://huggingface.co/docs/transformers/model_doc/phi3#transformers.Phi3Model)
* [SigLIP](https://huggingface.co/docs/transformers/model_doc/siglip)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
* [Qwen2](https://huggingface.co/docs/transformers/model_doc/qwen2#transformers.Qwen2Model)
* [Qwen2Audio](https://huggingface.co/docs/transformers/model_doc/qwen2_audio#transformers.Qwen2AudioEncoder)
* [Qwen2MoE](https://huggingface.co/docs/transformers/model_doc/qwen2_moe#transformers.Qwen2MoeModel)
* [Qwen2VL](https://huggingface.co/docs/transformers/model_doc/qwen2_vl#transformers.Qwen2VLModel)
* [Whisper](https://huggingface.


key: _node_content
string_value:
{"id_": "aa750ee1-c4ff-4197-8fc8-5e0d4552826e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c2df8d2a-7028-4ad3-9379-43e0506774a2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "ca0f5eb7741caf93c830e975b2042c83e9506d8beeed13d61135c341b2a84866", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0bff2c01-8569-40c6-adb5-6c84f029457d", "node_type": "1", "metadata": {}, "hash": "01b7ece2b090935049f47c8c9f39eb667f67a66d8d1b25276b7da1be2ef7e2b0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4060, "end_char_idx": 6503, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
MusicgenMelodyModel)
* [Nemotron](https://huggingface.co/docs/transformers/model_doc/nemotron)
* [NLLB](https://huggingface.co/docs/transformers/model_doc/nllb)
* [OLMo](https://huggingface.co/docs/transformers/model_doc/olmo#transformers.OlmoModel)
* [OPT](https://huggingface.co/docs/transformers/model_doc/opt#transformers.OPTModel)
* [Phi](https://huggingface.co/docs/transformers/model_doc/phi#transformers.PhiModel)
* [Phi3](https://huggingface.co/docs/transformers/model_doc/phi3#transformers.Phi3Model)
* [SigLIP](https://huggingface.co/docs/transformers/model_doc/siglip)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
* [Qwen2](https://huggingface.co/docs/transformers/model_doc/qwen2#transformers.Qwen2Model)
* [Qwen2Audio](https://huggingface.co/docs/transformers/model_doc/qwen2_audio#transformers.Qwen2AudioEncoder)
* [Qwen2MoE](https://huggingface.co/docs/transformers/model_doc/qwen2_moe#transformers.Qwen2MoeModel)
* [Qwen2VL](https://huggingface.co/docs/transformers/model_doc/qwen2_vl#transformers.Qwen2VLModel)
* [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperModel)
* [Wav2Vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.HubertModel)
* [data2vec_audio](https://huggingface.co/docs/transformers/main/en/model_doc/data2vec#transformers.Data2VecAudioModel)
* [Sew](https://huggingface.co/docs/transformers/main/en/model_doc/sew#transformers.SEWModel)
* [UniSpeech](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech#transformers.UniSpeechModel)
* [unispeech_sat](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech-sat#transformers.UniSpeechSatModel)

You can request to add FlashAttention-2 support for another model by opening a GitHub Issue or Pull Request.

Before you begin, make sure you have FlashAttention-2 installed.

<hfoptions id="install">
<hfoption id="NVIDIA">

```bash
pip install flash-attn --no-build-isolation
```

We strongly suggest referring to the detailed [installation instructions](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features) to learn more about supported hardware and data types!


key: _node_content
string_value:
{"id_": "0bff2c01-8569-40c6-adb5-6c84f029457d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa750ee1-c4ff-4197-8fc8-5e0d4552826e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "eb74660e50026ed96eb75427f49774c04bacb5905986ee147adc42a443b346e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4443492-bb17-4fd9-9471-ff3515188706", "node_type": "1", "metadata": {}, "hash": "2b03646c76fb23028f0bcde26f2ad2e40b1ad2163ee043c48c554fa4d3ae26a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5138, "end_char_idx": 7848, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Qwen2MoeModel)
* [Qwen2VL](https://huggingface.co/docs/transformers/model_doc/qwen2_vl#transformers.Qwen2VLModel)
* [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperModel)
* [Wav2Vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.HubertModel)
* [data2vec_audio](https://huggingface.co/docs/transformers/main/en/model_doc/data2vec#transformers.Data2VecAudioModel)
* [Sew](https://huggingface.co/docs/transformers/main/en/model_doc/sew#transformers.SEWModel)
* [UniSpeech](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech#transformers.UniSpeechModel)
* [unispeech_sat](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech-sat#transformers.UniSpeechSatModel)

You can request to add FlashAttention-2 support for another model by opening a GitHub Issue or Pull Request.

Before you begin, make sure you have FlashAttention-2 installed.

<hfoptions id="install">
<hfoption id="NVIDIA">

```bash
pip install flash-attn --no-build-isolation
```

We strongly suggest referring to the detailed [installation instructions](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features) to learn more about supported hardware and data types!

</hfoption>
<hfoption id="AMD">

FlashAttention-2 is also supported on AMD GPUs and current support is limited to **Instinct MI210**, **Instinct MI250** and **Instinct MI300**. We strongly suggest using this [Dockerfile](https://github.com/huggingface/optimum-amd/tree/main/docker/transformers-pytorch-amd-gpu-flash/Dockerfile) to use FlashAttention-2 on AMD GPUs.

</hfoption>
</hfoptions>

To enable FlashAttention-2, pass the argument `attn_implementation="flash_attention_2"` to [`~AutoModelForCausalLM.from_pretrained`]:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
```

<Tip>

FlashAttention-2 can only be used when the model's dtype is `fp16` or `bf16`. Make sure to cast your model to the appropriate dtype and load them on a supported device before using FlashAttention-2.

<br>

You can also set `use_flash_attention_2=True` to enable FlashAttention-2 but it is deprecated in favor of `attn_implementation="flash_attention_2"`.

</Tip>

FlashAttention-2 can be combined with other optimization techniques like quantization to further speedup inference.


key: _node_content
string_value:
{"id_": "e4443492-bb17-4fd9-9471-ff3515188706", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0bff2c01-8569-40c6-adb5-6c84f029457d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "f634b3cf21a6980c3565b1e0d6a7f11aaf2a20ff691635eff906e774aaa84f95", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c38792b5-2aed-4e5c-841b-4ebff5c0222a", "node_type": "1", "metadata": {}, "hash": "3329164b29608d66e7683702b3159761ea523b1a556afb98036610bf6198e2b4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6505, "end_char_idx": 9132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</hfoption>
<hfoption id="AMD">

FlashAttention-2 is also supported on AMD GPUs and current support is limited to **Instinct MI210**, **Instinct MI250** and **Instinct MI300**. We strongly suggest using this [Dockerfile](https://github.com/huggingface/optimum-amd/tree/main/docker/transformers-pytorch-amd-gpu-flash/Dockerfile) to use FlashAttention-2 on AMD GPUs.

</hfoption>
</hfoptions>

To enable FlashAttention-2, pass the argument `attn_implementation="flash_attention_2"` to [`~AutoModelForCausalLM.from_pretrained`]:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
```

<Tip>

FlashAttention-2 can only be used when the model's dtype is `fp16` or `bf16`. Make sure to cast your model to the appropriate dtype and load them on a supported device before using FlashAttention-2.

<br>

You can also set `use_flash_attention_2=True` to enable FlashAttention-2 but it is deprecated in favor of `attn_implementation="flash_attention_2"`.

</Tip>

FlashAttention-2 can be combined with other optimization techniques like quantization to further speedup inference. For example, you can combine FlashAttention-2 with 8-bit or 4-bit quantization:

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# load in 8bit
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_8bit=True,
    attn_implementation="flash_attention_2",
)

# load in 4bit
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    attn_implementation="flash_attention_2",
)
```

### Expected speedups

You can benefit from considerable speedups for inference, especially for inputs with long sequences. However, since FlashAttention-2 does not support computing attention scores with padding tokens, you must manually pad/unpad the attention scores for batched inference when the sequence contains padding tokens. This leads to a significant slowdown for batched generations with padding tokens.

To overcome this, you should use FlashAttention-2 without padding tokens in the sequence during training (by packing a dataset or [concatenating sequences](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py#L516) until reaching the maximum sequence length).


key: _node_content
string_value:
{"id_": "c38792b5-2aed-4e5c-841b-4ebff5c0222a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4443492-bb17-4fd9-9471-ff3515188706", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "91c701a7e0d46917003d34d930c9d69e6fb39271035231880ed8554373e59370", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b81548d4-9134-482c-869d-a1a8c81d110c", "node_type": "1", "metadata": {}, "hash": "021412bd0dd7de709897a2acfcfb1a710c3907684b4992499ef52384ec4955f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7447, "end_char_idx": 10751, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Make sure to cast your model to the appropriate dtype and load them on a supported device before using FlashAttention-2.

<br>

You can also set `use_flash_attention_2=True` to enable FlashAttention-2 but it is deprecated in favor of `attn_implementation="flash_attention_2"`.

</Tip>

FlashAttention-2 can be combined with other optimization techniques like quantization to further speedup inference. For example, you can combine FlashAttention-2 with 8-bit or 4-bit quantization:

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# load in 8bit
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_8bit=True,
    attn_implementation="flash_attention_2",
)

# load in 4bit
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    attn_implementation="flash_attention_2",
)
```

### Expected speedups

You can benefit from considerable speedups for inference, especially for inputs with long sequences. However, since FlashAttention-2 does not support computing attention scores with padding tokens, you must manually pad/unpad the attention scores for batched inference when the sequence contains padding tokens. This leads to a significant slowdown for batched generations with padding tokens.

To overcome this, you should use FlashAttention-2 without padding tokens in the sequence during training (by packing a dataset or [concatenating sequences](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py#L516) until reaching the maximum sequence length).

For a single forward pass on [tiiuae/falcon-7b](https://hf.co/tiiuae/falcon-7b) with a sequence length of 4096 and various batch sizes without padding tokens, the expected speedup is:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/falcon-7b-inference-large-seqlen.png">
</div>

For a single forward pass on [meta-llama/Llama-7b-hf](https://hf.co/meta-llama/Llama-7b-hf) with a sequence length of 4096 and various batch sizes without padding tokens, the expected speedup is:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-7b-inference-large-seqlen.png">
</div>

For sequences with padding tokens (generating with padding tokens), you need to unpad/pad the input sequences to correctly compute the attention scores. With a relatively small sequence length, a single forward pass creates overhead leading to a small speedup (in the example below, 30% of the input is filled with padding tokens):

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-2-small-seqlen-padding.png">
</div>

But for larger sequence lengths, you can expect even more speedup benefits:

<Tip>

FlashAttention is more memory efficient, meaning you can train on much larger sequence lengths without running into out-of-memory issues. You can potentially reduce memory usage up to 20x for larger sequence lengths. Take a look at the [flash-attention](https://github.com/Dao-AILab/flash-attention) repository for more details.


key: _node_content
string_value:
{"id_": "b81548d4-9134-482c-869d-a1a8c81d110c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c38792b5-2aed-4e5c-841b-4ebff5c0222a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9302e1516b4bbdfa77cece7f5bf946f525393f1ce6bb0a67fbfc5fdc7bfc579b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8404729e-0266-42a7-9233-77dbb93c92aa", "node_type": "1", "metadata": {}, "hash": "64b216d09d783aae60e7b812a523d8ea2808fcd2cf6a3bbeae6c6e3c4cea44e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9134, "end_char_idx": 12234, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For a single forward pass on [tiiuae/falcon-7b](https://hf.co/tiiuae/falcon-7b) with a sequence length of 4096 and various batch sizes without padding tokens, the expected speedup is:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/falcon-7b-inference-large-seqlen.png">
</div>

For a single forward pass on [meta-llama/Llama-7b-hf](https://hf.co/meta-llama/Llama-7b-hf) with a sequence length of 4096 and various batch sizes without padding tokens, the expected speedup is:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-7b-inference-large-seqlen.png">
</div>

For sequences with padding tokens (generating with padding tokens), you need to unpad/pad the input sequences to correctly compute the attention scores. With a relatively small sequence length, a single forward pass creates overhead leading to a small speedup (in the example below, 30% of the input is filled with padding tokens):

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-2-small-seqlen-padding.png">
</div>

But for larger sequence lengths, you can expect even more speedup benefits:

<Tip>

FlashAttention is more memory efficient, meaning you can train on much larger sequence lengths without running into out-of-memory issues. You can potentially reduce memory usage up to 20x for larger sequence lengths. Take a look at the [flash-attention](https://github.com/Dao-AILab/flash-attention) repository for more details.

</Tip>

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-2-large-seqlen-padding.png">
</div>

## PyTorch scaled dot product attention

PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood. SDPA support is currently being added natively in Transformers and is used by default for `torch>=2.1.1` when an implementation is available. You may also set `attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

For now, Transformers supports SDPA inference and training for the following architectures:
* [Audio Spectrogram Transformer](https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer#transformers.ASTModel)
* [Bart](https://huggingface.co/docs/transformers/model_doc/bart#transformers.BartModel)
* [Bert](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertModel)
* [Chameleon](https://huggingface.co/docs/transformers/model_doc/chameleon#transformers.Chameleon)
* [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPModel)
* [Cohere](https://huggingface.co/docs/transformers/model_doc/cohere#transformers.CohereModel)
* [Dbrx](https://huggingface.co/docs/transformers/model_doc/dbrx#transformers.


key: _node_content
string_value:
{"id_": "8404729e-0266-42a7-9233-77dbb93c92aa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b81548d4-9134-482c-869d-a1a8c81d110c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "c2d983df83f80150ee4971d17e34ab9ac63707622a22846f6e5a73423efb92f7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3be7135-b2a4-4046-89e8-1b6790e7e2bf", "node_type": "1", "metadata": {}, "hash": "b437404cc5070611097c856ab6150cc7431f0682d86da6678dcca5994ad5b2af", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10753, "end_char_idx": 13432, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-2-large-seqlen-padding.png">
</div>

## PyTorch scaled dot product attention

PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood. SDPA support is currently being added natively in Transformers and is used by default for `torch>=2.1.1` when an implementation is available. You may also set `attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

For now, Transformers supports SDPA inference and training for the following architectures:
* [Audio Spectrogram Transformer](https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer#transformers.ASTModel)
* [Bart](https://huggingface.co/docs/transformers/model_doc/bart#transformers.BartModel)
* [Bert](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertModel)
* [Chameleon](https://huggingface.co/docs/transformers/model_doc/chameleon#transformers.Chameleon)
* [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPModel)
* [Cohere](https://huggingface.co/docs/transformers/model_doc/cohere#transformers.CohereModel)
* [Dbrx](https://huggingface.co/docs/transformers/model_doc/dbrx#transformers.DbrxModel)
* [DeiT](https://huggingface.co/docs/transformers/model_doc/deit#transformers.DeiTModel)
* [Dpr](https://huggingface.co/docs/transformers/model_doc/dpr#transformers.DprReader)
* [Falcon](https://huggingface.co/docs/transformers/model_doc/falcon#transformers.FalconModel)
* [Gemma](https://huggingface.co/docs/transformers/model_doc/gemma#transformers.GemmaModel)
* [Gemma2](https://huggingface.co/docs/transformers/model_doc/gemma2#transformers.Gemma2Model)
* [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)
* [GPTBigCode](https://huggingface.co/docs/transformers/model_doc/gpt_bigcode#transformers.GPTBigCodeModel)
* [GPTNeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox#transformers.GPTNeoXModel)
* [JetMoe](https://huggingface.co/docs/transformers/model_doc/jetmoe#transformers.JetMoeModel)
* [Jamba](https://huggingface.co/docs/transformers/model_doc/jamba#transformers.JambaModel)
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [OLMo](https://huggingface.co/docs/transformers/model_doc/olmo#transformers.OlmoModel)
* [PaliGemma](https://huggingface.co/docs/transformers/model_doc/paligemma#transformers.


key: _node_content
string_value:
{"id_": "d3be7135-b2a4-4046-89e8-1b6790e7e2bf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8404729e-0266-42a7-9233-77dbb93c92aa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "fd7e77e16ac98b685a1432b2419acdbd39515804439a22d5d7ddc16731c7ff6c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ea9bcbf-a265-484e-965f-06486204c007", "node_type": "1", "metadata": {}, "hash": "0552d18a9a088900be4334a6b20f9888679892a332ea562db622a6e2ca0d2dfc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12185, "end_char_idx": 14578, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/dbrx#transformers.DbrxModel)
* [DeiT](https://huggingface.co/docs/transformers/model_doc/deit#transformers.DeiTModel)
* [Dpr](https://huggingface.co/docs/transformers/model_doc/dpr#transformers.DprReader)
* [Falcon](https://huggingface.co/docs/transformers/model_doc/falcon#transformers.FalconModel)
* [Gemma](https://huggingface.co/docs/transformers/model_doc/gemma#transformers.GemmaModel)
* [Gemma2](https://huggingface.co/docs/transformers/model_doc/gemma2#transformers.Gemma2Model)
* [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)
* [GPTBigCode](https://huggingface.co/docs/transformers/model_doc/gpt_bigcode#transformers.GPTBigCodeModel)
* [GPTNeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox#transformers.GPTNeoXModel)
* [JetMoe](https://huggingface.co/docs/transformers/model_doc/jetmoe#transformers.JetMoeModel)
* [Jamba](https://huggingface.co/docs/transformers/model_doc/jamba#transformers.JambaModel)
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [OLMo](https://huggingface.co/docs/transformers/model_doc/olmo#transformers.OlmoModel)
* [PaliGemma](https://huggingface.co/docs/transformers/model_doc/paligemma#transformers.PaliGemmaForConditionalGeneration)
* [Phi](https://huggingface.co/docs/transformers/model_doc/phi#transformers.PhiModel)
* [Phi3](https://huggingface.co/docs/transformers/model_doc/phi3#transformers.Phi3Model)
* [Idefics](https://huggingface.co/docs/transformers/model_doc/idefics#transformers.IdeficsModel)
* [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperModel)
* [Mistral](https://huggingface.co/docs/transformers/model_doc/mistral#transformers.MistralModel)
* [Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral#transformers.MixtralModel)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
* [Qwen2](https://huggingface.co/docs/transformers/model_doc/qwen2#transformers.Qwen2Model)
* [Qwen2Audio](https://huggingface.co/docs/transformers/model_doc/qwen2_audio#transformers.Qwen2AudioEncoder)
* [Qwen2MoE](https://huggingface.co/docs/transformers/model_doc/qwen2_moe#transformers.Qwen2MoeModel)
* [Qwen2VL](https://huggingface.


key: _node_content
string_value:
{"id_": "2ea9bcbf-a265-484e-965f-06486204c007", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d3be7135-b2a4-4046-89e8-1b6790e7e2bf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "1f075a8aa8b8ef814ba0aac07d6fff9d22d48d9eca10923f19d3f851ed713275", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ab864342-2b7a-4139-a7a2-8cab8d5b4d6b", "node_type": "1", "metadata": {}, "hash": "a427202087ff194cc31264089bef6bf08f21b9a3e40bc93d14949a2653e8d944", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13378, "end_char_idx": 15773, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/paligemma#transformers.PaliGemmaForConditionalGeneration)
* [Phi](https://huggingface.co/docs/transformers/model_doc/phi#transformers.PhiModel)
* [Phi3](https://huggingface.co/docs/transformers/model_doc/phi3#transformers.Phi3Model)
* [Idefics](https://huggingface.co/docs/transformers/model_doc/idefics#transformers.IdeficsModel)
* [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperModel)
* [Mistral](https://huggingface.co/docs/transformers/model_doc/mistral#transformers.MistralModel)
* [Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral#transformers.MixtralModel)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
* [Qwen2](https://huggingface.co/docs/transformers/model_doc/qwen2#transformers.Qwen2Model)
* [Qwen2Audio](https://huggingface.co/docs/transformers/model_doc/qwen2_audio#transformers.Qwen2AudioEncoder)
* [Qwen2MoE](https://huggingface.co/docs/transformers/model_doc/qwen2_moe#transformers.Qwen2MoeModel)
* [Qwen2VL](https://huggingface.co/docs/transformers/model_doc/qwen2_vl#transformers.Qwen2VLModel)
* [Musicgen](https://huggingface.co/docs/transformers/model_doc/musicgen#transformers.MusicgenModel)
* [MusicGen Melody](https://huggingface.co/docs/transformers/model_doc/musicgen_melody#transformers.MusicgenMelodyModel)
* [Nemotron](https://huggingface.co/docs/transformers/model_doc/nemotron)
* [ViT](https://huggingface.co/docs/transformers/model_doc/vit#transformers.ViTModel)
* [ViTHybrid](https://huggingface.co/docs/transformers/model_doc/vit_hybrid#transformers.ViTHybridModel)
* [ViTMAE](https://huggingface.co/docs/transformers/model_doc/vit_mae#transformers.ViTMAEModel)
* [ViTMSN](https://huggingface.co/docs/transformers/model_doc/vit_msn#transformers.ViTMSNModel)
* [VideoMAE](https://huggingface.co/docs/transformers/model_doc/videomae#transformers.VideoMAEModell)
* [wav2vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.HubertModel)
* [data2vec_audio](https://huggingface.co/docs/transformers/main/en/model_doc/data2vec#transformers.Data2VecAudioModel)
* [SigLIP](https://huggingface.


key: _node_content
string_value:
{"id_": "ab864342-2b7a-4139-a7a2-8cab8d5b4d6b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ea9bcbf-a265-484e-965f-06486204c007", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "e4eff9ddbae31922d6f353973824646a5e71011e7f51981d3f6f15a91039a78f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7ac39afa-b729-4d7b-ad49-fbf7f8e820f6", "node_type": "1", "metadata": {}, "hash": "c34720750aefc53f2a4dca785e4375a812cabbc86663d4d9c416554d5e05f1f4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14531, "end_char_idx": 16696, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Qwen2MoeModel)
* [Qwen2VL](https://huggingface.co/docs/transformers/model_doc/qwen2_vl#transformers.Qwen2VLModel)
* [Musicgen](https://huggingface.co/docs/transformers/model_doc/musicgen#transformers.MusicgenModel)
* [MusicGen Melody](https://huggingface.co/docs/transformers/model_doc/musicgen_melody#transformers.MusicgenMelodyModel)
* [Nemotron](https://huggingface.co/docs/transformers/model_doc/nemotron)
* [ViT](https://huggingface.co/docs/transformers/model_doc/vit#transformers.ViTModel)
* [ViTHybrid](https://huggingface.co/docs/transformers/model_doc/vit_hybrid#transformers.ViTHybridModel)
* [ViTMAE](https://huggingface.co/docs/transformers/model_doc/vit_mae#transformers.ViTMAEModel)
* [ViTMSN](https://huggingface.co/docs/transformers/model_doc/vit_msn#transformers.ViTMSNModel)
* [VideoMAE](https://huggingface.co/docs/transformers/model_doc/videomae#transformers.VideoMAEModell)
* [wav2vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.HubertModel)
* [data2vec_audio](https://huggingface.co/docs/transformers/main/en/model_doc/data2vec#transformers.Data2VecAudioModel)
* [SigLIP](https://huggingface.co/docs/transformers/model_doc/siglip)
* [Sew](https://huggingface.co/docs/transformers/main/en/model_doc/sew#transformers.SEWModel)
* [UniSpeech](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech#transformers.UniSpeechModel)
* [unispeech_sat](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech-sat#transformers.UniSpeechSatModel)
* [YOLOS](https://huggingface.co/docs/transformers/model_doc/yolos#transformers.YolosModel)


<Tip>

FlashAttention can only be used for models with the `fp16` or `bf16` torch type, so make sure to cast your model to the appropriate type first. The memory-efficient attention backend is able to handle `fp32` models.

</Tip>

<Tip>

SDPA does not support certain sets of attention parameters, such as `head_mask` and `output_attentions=True`.
In that case, you should see a warning message and we will fall back to the (slower) eager implementation.


key: _node_content
string_value:
{"id_": "7ac39afa-b729-4d7b-ad49-fbf7f8e820f6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ab864342-2b7a-4139-a7a2-8cab8d5b4d6b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "98ff72426e3260f5d0358244cf1e9779cabc027515ac8b74619f19bd5ebc102a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9e71fded-a859-469c-936a-1614ad709d5d", "node_type": "1", "metadata": {}, "hash": "64a4589c5365e2cffc9275a0dc5fb491bf9e06b34df55e674b8ae9408b016fe3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15357, "end_char_idx": 18270, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/videomae#transformers.VideoMAEModell)
* [wav2vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.HubertModel)
* [data2vec_audio](https://huggingface.co/docs/transformers/main/en/model_doc/data2vec#transformers.Data2VecAudioModel)
* [SigLIP](https://huggingface.co/docs/transformers/model_doc/siglip)
* [Sew](https://huggingface.co/docs/transformers/main/en/model_doc/sew#transformers.SEWModel)
* [UniSpeech](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech#transformers.UniSpeechModel)
* [unispeech_sat](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech-sat#transformers.UniSpeechSatModel)
* [YOLOS](https://huggingface.co/docs/transformers/model_doc/yolos#transformers.YolosModel)


<Tip>

FlashAttention can only be used for models with the `fp16` or `bf16` torch type, so make sure to cast your model to the appropriate type first. The memory-efficient attention backend is able to handle `fp32` models.

</Tip>

<Tip>

SDPA does not support certain sets of attention parameters, such as `head_mask` and `output_attentions=True`.
In that case, you should see a warning message and we will fall back to the (slower) eager implementation.

</Tip>

By default, SDPA selects the most performant kernel available but you can check whether a backend is available in a given setting (hardware, problem size) with [`torch.backends.cuda.sdp_kernel`](https://pytorch.org/docs/master/backends.html#torch.backends.cuda.sdp_kernel) as a context manager:

```diff
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", torch_dtype=torch.float16).to("cuda")

input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")

+ with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

If you see a bug with the traceback below, try using the nightly version of PyTorch which may have broader coverage for FlashAttention:

```bash
RuntimeError: No available kernel. Aborting execution.

# install PyTorch nightly
pip3 install -U --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118
```

## BetterTransformer

<Tip warning={true}>

Some BetterTransformer features are being upstreamed to Transformers with default support for native `torch.nn.scaled_dot_product_attention`. BetterTransformer still has a wider coverage than the Transformers SDPA integration, but you can expect more and more architectures to natively support SDPA in Transformers.


key: _node_content
string_value:
{"id_": "9e71fded-a859-469c-936a-1614ad709d5d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7ac39afa-b729-4d7b-ad49-fbf7f8e820f6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9270e5562f160efcc3e2c7e13dc33af7b3457f91d1ddaa950ab987f893070525", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "75d49e06-3e23-4309-8478-bb714015f318", "node_type": "1", "metadata": {}, "hash": "00c96c886c7f85e488f593363bf3b14d05feced924910e2dab5ec6cce564d6e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16590, "end_char_idx": 20137, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In that case, you should see a warning message and we will fall back to the (slower) eager implementation.

</Tip>

By default, SDPA selects the most performant kernel available but you can check whether a backend is available in a given setting (hardware, problem size) with [`torch.backends.cuda.sdp_kernel`](https://pytorch.org/docs/master/backends.html#torch.backends.cuda.sdp_kernel) as a context manager:

```diff
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", torch_dtype=torch.float16).to("cuda")

input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")

+ with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

If you see a bug with the traceback below, try using the nightly version of PyTorch which may have broader coverage for FlashAttention:

```bash
RuntimeError: No available kernel. Aborting execution.

# install PyTorch nightly
pip3 install -U --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118
```

## BetterTransformer

<Tip warning={true}>

Some BetterTransformer features are being upstreamed to Transformers with default support for native `torch.nn.scaled_dot_product_attention`. BetterTransformer still has a wider coverage than the Transformers SDPA integration, but you can expect more and more architectures to natively support SDPA in Transformers.

</Tip>

<Tip>

Check out our benchmarks with BetterTransformer and scaled dot product attention in the [Out of the box acceleration and memory savings of 🤗 decoder models with PyTorch 2.0](https://pytorch.org/blog/out-of-the-box-acceleration/) and learn more about the fastpath execution in the [BetterTransformer](https://medium.com/pytorch/bettertransformer-out-of-the-box-performance-for-huggingface-transformers-3fbe27d50ab2) blog post.

</Tip>

BetterTransformer accelerates inference with its fastpath (native PyTorch specialized implementation of Transformer functions) execution. The two optimizations in the fastpath execution are:

1. fusion, which combines multiple sequential operations into a single "kernel" to reduce the number of computation steps
2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors

BetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention (SDPA)](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention), and it calls optimized kernels like [FlashAttention](https://huggingface.co/papers/2205.14135) under the hood.

Before you start, make sure you have 🤗 Optimum [installed](https://huggingface.co/docs/optimum/installation).

Then you can enable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:

```python
model = model.to_bettertransformer()
```

You can return the original Transformers model with the [`~PreTrainedModel.reverse_bettertransformer`] method. You should use this before saving your model to use the canonical Transformers modeling:

```py
model = model.reverse_bettertransformer()
model.save_pretrained("saved_model")
```

## bitsandbytes

bitsandbytes is a quantization library that includes support for 4-bit and 8-bit quantization.


key: _node_content
string_value:
{"id_": "75d49e06-3e23-4309-8478-bb714015f318", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9e71fded-a859-469c-936a-1614ad709d5d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "75789e601d8fed66cd86c7e60eaa01c6f50f67aa3561e6e08950985b1c823df7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c220fac-6cdb-44e1-b919-cb290625ebef", "node_type": "1", "metadata": {}, "hash": "15a05defd441bde2c76c492113669685e575ad50b4b4605dc3b7377dc88771ee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18714, "end_char_idx": 21859, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

BetterTransformer accelerates inference with its fastpath (native PyTorch specialized implementation of Transformer functions) execution. The two optimizations in the fastpath execution are:

1. fusion, which combines multiple sequential operations into a single "kernel" to reduce the number of computation steps
2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors

BetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention (SDPA)](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention), and it calls optimized kernels like [FlashAttention](https://huggingface.co/papers/2205.14135) under the hood.

Before you start, make sure you have 🤗 Optimum [installed](https://huggingface.co/docs/optimum/installation).

Then you can enable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:

```python
model = model.to_bettertransformer()
```

You can return the original Transformers model with the [`~PreTrainedModel.reverse_bettertransformer`] method. You should use this before saving your model to use the canonical Transformers modeling:

```py
model = model.reverse_bettertransformer()
model.save_pretrained("saved_model")
```

## bitsandbytes

bitsandbytes is a quantization library that includes support for 4-bit and 8-bit quantization. Quantization reduces your model size compared to its native full precision version, making it easier to fit large models onto GPUs with limited memory.

Make sure you have bitsandbytes and 🤗 Accelerate installed:

```bash
# these versions support 8-bit and 4-bit
pip install bitsandbytes>=0.39.0 accelerate>=0.20.0

# install Transformers
pip install transformers
```

### 4-bit

To load a model in 4-bit for inference, use the `load_in_4bit` parameter. The `device_map` parameter is optional, but we recommend setting it to `"auto"` to allow 🤗 Accelerate to automatically and efficiently allocate the model given the available resources in the environment.

```py
from transformers import AutoModelForCausalLM

model_name = "bigscience/bloom-2b5"
model_4bit = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_4bit=True)
```

To load a model in 4-bit for inference with multiple GPUs, you can control how much GPU RAM you want to allocate to each GPU. For example, to distribute 600MB of memory to the first GPU and 1GB of memory to the second GPU:

```py
max_memory_mapping = {0: "600MB", 1: "1GB"}
model_name = "bigscience/bloom-3b"
model_4bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_4bit=True, max_memory=max_memory_mapping
)
```

### 8-bit

<Tip>

If you're curious and interested in learning more about the concepts underlying 8-bit quantization, read the [Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes](https://huggingface.co/blog/hf-bitsandbytes-integration) blog post.

</Tip>

To load a model in 8-bit for inference, use the `load_in_8bit` parameter.


key: _node_content
string_value:
{"id_": "3c220fac-6cdb-44e1-b919-cb290625ebef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "75d49e06-3e23-4309-8478-bb714015f318", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "f820deba41631c0222895c433e3a10261ac898e707d1fc3c372615e9ad6a8856", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b4a3115e-8456-4809-8b28-f703c5f3fe92", "node_type": "1", "metadata": {}, "hash": "39d7cf5d1af214a00d8b78dfe1c506987d0bbc3c4b5d3287bd66e8c22eed2595", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20592, "end_char_idx": 23338, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `device_map` parameter is optional, but we recommend setting it to `"auto"` to allow 🤗 Accelerate to automatically and efficiently allocate the model given the available resources in the environment.

```py
from transformers import AutoModelForCausalLM

model_name = "bigscience/bloom-2b5"
model_4bit = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_4bit=True)
```

To load a model in 4-bit for inference with multiple GPUs, you can control how much GPU RAM you want to allocate to each GPU. For example, to distribute 600MB of memory to the first GPU and 1GB of memory to the second GPU:

```py
max_memory_mapping = {0: "600MB", 1: "1GB"}
model_name = "bigscience/bloom-3b"
model_4bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_4bit=True, max_memory=max_memory_mapping
)
```

### 8-bit

<Tip>

If you're curious and interested in learning more about the concepts underlying 8-bit quantization, read the [Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes](https://huggingface.co/blog/hf-bitsandbytes-integration) blog post.

</Tip>

To load a model in 8-bit for inference, use the `load_in_8bit` parameter. The `device_map` parameter is optional, but we recommend setting it to `"auto"` to allow 🤗 Accelerate to automatically and efficiently allocate the model given the available resources in the environment:

```py
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model_name = "bigscience/bloom-2b5"
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=BitsAndBytesConfig(load_in_8bit=True))
```

If you're loading a model in 8-bit for text generation, you should use the [`~transformers.GenerationMixin.generate`] method instead of the [`Pipeline`] function which is not optimized for 8-bit models and will be slower. Some sampling strategies, like nucleus sampling, are also not supported by the [`Pipeline`] for 8-bit models. You should also place all inputs on the same device as the model:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "bigscience/bloom-2b5"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=BitsAndBytesConfig(load_in_8bit=True))

prompt = "Hello, my llama is cute"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
generated_ids = model.generate(**inputs)
outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
```

To load a model in 4-bit for inference with multiple GPUs, you can control how much GPU RAM you want to allocate to each GPU.


key: _node_content
string_value:
{"id_": "b4a3115e-8456-4809-8b28-f703c5f3fe92", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c220fac-6cdb-44e1-b919-cb290625ebef", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "52d34ac199e41d736525b26a537e0fefba4f4cef3ba06c5e6ec8861c67715901", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6a48f73c-90df-430c-bd07-be394955b3b4", "node_type": "1", "metadata": {}, "hash": "895f2edd7d2c13665ce49c60128285f3796008c3f1afb6469f60dda0c892ac13", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21778, "end_char_idx": 24614, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

To load a model in 8-bit for inference, use the `load_in_8bit` parameter. The `device_map` parameter is optional, but we recommend setting it to `"auto"` to allow 🤗 Accelerate to automatically and efficiently allocate the model given the available resources in the environment:

```py
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model_name = "bigscience/bloom-2b5"
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=BitsAndBytesConfig(load_in_8bit=True))
```

If you're loading a model in 8-bit for text generation, you should use the [`~transformers.GenerationMixin.generate`] method instead of the [`Pipeline`] function which is not optimized for 8-bit models and will be slower. Some sampling strategies, like nucleus sampling, are also not supported by the [`Pipeline`] for 8-bit models. You should also place all inputs on the same device as the model:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "bigscience/bloom-2b5"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=BitsAndBytesConfig(load_in_8bit=True))

prompt = "Hello, my llama is cute"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
generated_ids = model.generate(**inputs)
outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
```

To load a model in 4-bit for inference with multiple GPUs, you can control how much GPU RAM you want to allocate to each GPU. For example, to distribute 1GB of memory to the first GPU and 2GB of memory to the second GPU:

```py
max_memory_mapping = {0: "1GB", 1: "2GB"}
model_name = "bigscience/bloom-3b"
model_8bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_8bit=True, max_memory=max_memory_mapping
)
```

<Tip>

Feel free to try running a 11 billion parameter [T5 model](https://colab.research.google.com/drive/1YORPWx4okIHXnjW7MSAidXN29mPVNT7F?usp=sharing) or the 3 billion parameter [BLOOM model](https://colab.research.google.com/drive/1qOjXfQIAULfKvZqwCen8-MoWKGdSatZ4?usp=sharing) for inference on Google Colab's free tier GPUs!

</Tip>

## 🤗 Optimum

<Tip>

Learn more details about using ORT with 🤗 Optimum in the [Accelerated inference on NVIDIA GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#accelerated-inference-on-nvidia-gpus) and [Accelerated inference on AMD GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu#accelerated-inference-on-amd-gpus) guides. This section only provides a brief and simple example.

</Tip>

ONNX Runtime (ORT) is a model accelerator that supports accelerated inference on Nvidia GPUs, and AMD GPUs that use [ROCm](https://www.amd.com/en/products/software/rocm.html) stack.


key: _node_content
string_value:
{"id_": "6a48f73c-90df-430c-bd07-be394955b3b4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b4a3115e-8456-4809-8b28-f703c5f3fe92", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "f61df08f36906633c1136352d7f66ea6335e3216e6a3ce5e24c28e56bae17b6f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "775ee1e1-4f94-4dfc-8bd8-fbccf5e87241", "node_type": "1", "metadata": {}, "hash": "4fb7fa09c09665776357c1f165b687645b85ad0ae4e9cc8bc300cb83ee1c2ff0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23339, "end_char_idx": 25503, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, to distribute 1GB of memory to the first GPU and 2GB of memory to the second GPU:

```py
max_memory_mapping = {0: "1GB", 1: "2GB"}
model_name = "bigscience/bloom-3b"
model_8bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_8bit=True, max_memory=max_memory_mapping
)
```

<Tip>

Feel free to try running a 11 billion parameter [T5 model](https://colab.research.google.com/drive/1YORPWx4okIHXnjW7MSAidXN29mPVNT7F?usp=sharing) or the 3 billion parameter [BLOOM model](https://colab.research.google.com/drive/1qOjXfQIAULfKvZqwCen8-MoWKGdSatZ4?usp=sharing) for inference on Google Colab's free tier GPUs!

</Tip>

## 🤗 Optimum

<Tip>

Learn more details about using ORT with 🤗 Optimum in the [Accelerated inference on NVIDIA GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#accelerated-inference-on-nvidia-gpus) and [Accelerated inference on AMD GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu#accelerated-inference-on-amd-gpus) guides. This section only provides a brief and simple example.

</Tip>

ONNX Runtime (ORT) is a model accelerator that supports accelerated inference on Nvidia GPUs, and AMD GPUs that use [ROCm](https://www.amd.com/en/products/software/rocm.html) stack. ORT uses optimization techniques like fusing common operations into a single node and constant folding to reduce the number of computations performed and speedup inference. ORT also places the most computationally intensive operations on the GPU and the rest on the CPU to intelligently distribute the workload between the two devices.

ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers. You'll need to use an [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and specify the `provider` parameter which can be set to either [`CUDAExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#cudaexecutionprovider), [`ROCMExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu) or [`TensorrtExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#tensorrtexecutionprovider).


key: _node_content
string_value:
{"id_": "775ee1e1-4f94-4dfc-8bd8-fbccf5e87241", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6a48f73c-90df-430c-bd07-be394955b3b4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "a22ee9204526c55a30f35d69b52e26ac97aad75cecebfa8531e54d19a707b61a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dcefcf4c-2950-47e8-972e-4d69342131ec", "node_type": "1", "metadata": {}, "hash": "6983403900b43b4cde8e5634c3937a5cb5a14610c1a42b3f450fc8fa4bf4251a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23990, "end_char_idx": 26530, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## 🤗 Optimum

<Tip>

Learn more details about using ORT with 🤗 Optimum in the [Accelerated inference on NVIDIA GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#accelerated-inference-on-nvidia-gpus) and [Accelerated inference on AMD GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu#accelerated-inference-on-amd-gpus) guides. This section only provides a brief and simple example.

</Tip>

ONNX Runtime (ORT) is a model accelerator that supports accelerated inference on Nvidia GPUs, and AMD GPUs that use [ROCm](https://www.amd.com/en/products/software/rocm.html) stack. ORT uses optimization techniques like fusing common operations into a single node and constant folding to reduce the number of computations performed and speedup inference. ORT also places the most computationally intensive operations on the GPU and the rest on the CPU to intelligently distribute the workload between the two devices.

ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers. You'll need to use an [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and specify the `provider` parameter which can be set to either [`CUDAExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#cudaexecutionprovider), [`ROCMExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu) or [`TensorrtExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#tensorrtexecutionprovider). If you want to load a model that was not yet exported to ONNX, you can set `export=True` to convert your model on-the-fly to the ONNX format:

```py
from optimum.onnxruntime import ORTModelForSequenceClassification

ort_model = ORTModelForSequenceClassification.from_pretrained(
  "distilbert/distilbert-base-uncased-finetuned-sst-2-english",
  export=True,
  provider="CUDAExecutionProvider",
)
```

Now you're free to use the model for inference:

```py
from optimum.pipelines import pipeline
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased-finetuned-sst-2-english")

pipeline = pipeline(task="text-classification", model=ort_model, tokenizer=tokenizer, device="cuda:0")
result = pipeline("Both the music and visual were astounding, not to mention the actors performance.")
```

## Combine optimizations

It is often possible to combine several of the optimization techniques described above to get the best inference performance possible for your model.


key: _node_content
string_value:
{"id_": "dcefcf4c-2950-47e8-972e-4d69342131ec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3af22dbf-d566-4ef7-8034-84cbf2a07850", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "9ab6187133ff4cf61b86f445d5d8fef44bd89a0ad3ec7e01ff7149edf99f2303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "775ee1e1-4f94-4dfc-8bd8-fbccf5e87241", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7227, "retrieve_doc": true, "source": "transformers"}, "hash": "1922a9164bf87289112b121c2dc94f4306114d518b846d897ab25ca5a96b2fe1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24952, "end_char_idx": 27435, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers. You'll need to use an [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and specify the `provider` parameter which can be set to either [`CUDAExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#cudaexecutionprovider), [`ROCMExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu) or [`TensorrtExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#tensorrtexecutionprovider). If you want to load a model that was not yet exported to ONNX, you can set `export=True` to convert your model on-the-fly to the ONNX format:

```py
from optimum.onnxruntime import ORTModelForSequenceClassification

ort_model = ORTModelForSequenceClassification.from_pretrained(
  "distilbert/distilbert-base-uncased-finetuned-sst-2-english",
  export=True,
  provider="CUDAExecutionProvider",
)
```

Now you're free to use the model for inference:

```py
from optimum.pipelines import pipeline
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased-finetuned-sst-2-english")

pipeline = pipeline(task="text-classification", model=ort_model, tokenizer=tokenizer, device="cuda:0")
result = pipeline("Both the music and visual were astounding, not to mention the actors performance.")
```

## Combine optimizations

It is often possible to combine several of the optimization techniques described above to get the best inference performance possible for your model. For example, you can load a model in 4-bit, and then enable BetterTransformer with FlashAttention:

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# load model in 4-bit
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", quantization_config=quantization_config)

# enable BetterTransformer
model = model.to_bettertransformer()

input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")

# enable FlashAttention
with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```


key: _node_content
string_value:
{"id_": "5646e085-fb72-44dd-ae7b-df286d64f7fc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0a976eb2-9650-4638-be27-0840e80eb0c4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "90bf510721ec54ac392e192edee32d70442722de8c43e41a564ce98d29812cda", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2227587a-0150-46ef-8eee-7c3a09ef77b9", "node_type": "1", "metadata": {}, "hash": "9ccfbd280615fea0c7e9ac62bd7ee5551db95b730ecddcfac28f222f8a580983", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2738, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Efficient Training on Multiple CPUs

When training on a single CPU is too slow, we can use multiple CPUs. This guide focuses on PyTorch-based DDP enabling
distributed CPU training efficiently on [bare metal](#usage-in-trainer) and [Kubernetes](#usage-with-kubernetes).

## Intel® oneCCL Bindings for PyTorch

[Intel® oneCCL](https://github.com/oneapi-src/oneCCL) (collective communications library) is a library for efficient distributed deep learning training implementing such collectives like allreduce, allgather, alltoall. For more information on oneCCL, please refer to the [oneCCL documentation](https://spec.oneapi.com/versions/latest/elements/oneCCL/source/index.html) and [oneCCL specification](https://spec.oneapi.com/versions/latest/elements/oneCCL/source/index.html).

Module `oneccl_bindings_for_pytorch` (`torch_ccl` before version 1.12)  implements PyTorch C10D ProcessGroup API and can be dynamically loaded as external ProcessGroup and only works on Linux platform now

Check more detailed information for [oneccl_bind_pt](https://github.com/intel/torch-ccl).

### Intel® oneCCL Bindings for PyTorch installation

Wheel files are available for the following Python versions:

| Extension Version | Python 3.6 | Python 3.7 | Python 3.8 | Python 3.9 | Python 3.10 |
| :---------------: | :--------: | :--------: | :--------: | :--------: | :---------: |
| 2.1.0             |            | √          | √          | √          | √           |
| 2.0.0             |            | √          | √          | √          | √           |
| 1.13.0            |            | √          | √          | √          | √           |
| 1.12.100          |            | √          | √          | √          | √           |
| 1.12.0            |            | √          | √          | √          | √           |

Please run `pip list | grep torch` to get your `pytorch_version`.
```bash
pip install oneccl_bind_pt=={pytorch_version} -f https://developer.intel.com/ipex-whl-stable-cpu
```
where `{pytorch_version}` should be your PyTorch version, for instance 2.1.0.
Check more approaches for [oneccl_bind_pt installation](https://github.com/intel/torch-ccl).
Versions of oneCCL and PyTorch must match.

<Tip warning={true}>

oneccl_bindings_for_pytorch 1.12.0 prebuilt wheel does not work with PyTorch 1.12.1 (it is for PyTorch 1.12.0)
PyTorch 1.12.1 should work with oneccl_bindings_for_pytorch 1.12.100

</Tip>

## Intel® MPI library
Use this standards-based MPI implementation to deliver flexible, efficient, scalable cluster messaging on Intel® architecture. This component is part of the Intel® oneAPI HPC Toolkit.

oneccl_bindings_for_pytorch is installed along with the MPI tool set. Need to source the environment before using it.


key: _node_content
string_value:
{"id_": "2227587a-0150-46ef-8eee-7c3a09ef77b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0a976eb2-9650-4638-be27-0840e80eb0c4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "90bf510721ec54ac392e192edee32d70442722de8c43e41a564ce98d29812cda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5646e085-fb72-44dd-ae7b-df286d64f7fc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "bafb8174e88bbfddc6cadb16953124a197a232605fb00ba122b998a09be94dae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e5c2e21-d17e-4b55-a85d-11fd4bbfe25c", "node_type": "1", "metadata": {}, "hash": "cc42a1766cb4ac5780371b8b51f5518e1f7b69ad5217ba383aec1bfae4ddac46", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1879, "end_char_idx": 4762, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```bash
pip install oneccl_bind_pt=={pytorch_version} -f https://developer.intel.com/ipex-whl-stable-cpu
```
where `{pytorch_version}` should be your PyTorch version, for instance 2.1.0.
Check more approaches for [oneccl_bind_pt installation](https://github.com/intel/torch-ccl).
Versions of oneCCL and PyTorch must match.

<Tip warning={true}>

oneccl_bindings_for_pytorch 1.12.0 prebuilt wheel does not work with PyTorch 1.12.1 (it is for PyTorch 1.12.0)
PyTorch 1.12.1 should work with oneccl_bindings_for_pytorch 1.12.100

</Tip>

## Intel® MPI library
Use this standards-based MPI implementation to deliver flexible, efficient, scalable cluster messaging on Intel® architecture. This component is part of the Intel® oneAPI HPC Toolkit.

oneccl_bindings_for_pytorch is installed along with the MPI tool set. Need to source the environment before using it.

for Intel® oneCCL >= 1.12.0
```bash
oneccl_bindings_for_pytorch_path=$(python -c "from oneccl_bindings_for_pytorch import cwd; print(cwd)")
source $oneccl_bindings_for_pytorch_path/env/setvars.sh
```

for Intel® oneCCL whose version < 1.12.0
```bash
torch_ccl_path=$(python -c "import torch; import torch_ccl; import os;  print(os.path.abspath(os.path.dirname(torch_ccl.__file__)))")
source $torch_ccl_path/env/setvars.sh
```

#### Intel® Extension for PyTorch installation

Intel Extension for PyTorch (IPEX) provides performance optimizations for CPU training with both Float32 and BFloat16 (refer to the [single CPU section](./perf_train_cpu) to learn more).


The following "Usage in Trainer" takes mpirun in Intel® MPI library as an example.


## Usage in Trainer
To enable multi CPU distributed training in the Trainer with the ccl backend, users should add **`--ddp_backend ccl`** in the command arguments.

Let's see an example with the [question-answering example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering)


The following command enables training with 2 processes on one Xeon node, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.
```shell script
 export CCL_WORKER_COUNT=1
 export MASTER_ADDR=127.0.0.1
 mpirun -n 2 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path google-bert/bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex
```
The following command enables training with a total of four processes on two Xeons (node0 and node1, taking node0 as the main process), ppn (processes per node) is set to 2, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.


key: _node_content
string_value:
{"id_": "7e5c2e21-d17e-4b55-a85d-11fd4bbfe25c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0a976eb2-9650-4638-be27-0840e80eb0c4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "90bf510721ec54ac392e192edee32d70442722de8c43e41a564ce98d29812cda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2227587a-0150-46ef-8eee-7c3a09ef77b9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "d340ba85ea319c283961145e611f9782806746e9e14948582422ab305ed99fd5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4e5dc914-578b-4a4d-868a-c894ebf19509", "node_type": "1", "metadata": {}, "hash": "68ae3027c0bb5c2d3a60e6ab6c18dd5847021e92b8559ebe603852bb4b0cba5b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3404, "end_char_idx": 6320, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The following "Usage in Trainer" takes mpirun in Intel® MPI library as an example.


## Usage in Trainer
To enable multi CPU distributed training in the Trainer with the ccl backend, users should add **`--ddp_backend ccl`** in the command arguments.

Let's see an example with the [question-answering example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering)


The following command enables training with 2 processes on one Xeon node, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.
```shell script
 export CCL_WORKER_COUNT=1
 export MASTER_ADDR=127.0.0.1
 mpirun -n 2 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path google-bert/bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex
```
The following command enables training with a total of four processes on two Xeons (node0 and node1, taking node0 as the main process), ppn (processes per node) is set to 2, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.

In node0, you need to create a configuration file which contains the IP addresses of each node (for example hostfile) and pass that configuration file path as an argument.
```shell script
 cat hostfile
 xxx.xxx.xxx.xxx #node0 ip
 xxx.xxx.xxx.xxx #node1 ip
```
Now, run the following command in node0 and **4DDP** will be enabled in node0 and node1 with BF16 auto mixed precision:
```shell script
 export CCL_WORKER_COUNT=1
 export MASTER_ADDR=xxx.xxx.xxx.xxx #node0 ip
 mpirun -f hostfile -n 4 -ppn 2 \
 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path google-bert/bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex \
 --bf16
```

## Usage with Kubernetes

The same distributed training job from the previous section can be deployed to a Kubernetes cluster using the
[Kubeflow PyTorchJob training operator](https://www.kubeflow.org/docs/components/training/pytorch/).

### Setup

This example assumes that you have:
* Access to a Kubernetes cluster with [Kubeflow installed](https://www.kubeflow.org/docs/started/installing-kubeflow/)
* [`kubectl`](https://kubernetes.io/docs/tasks/tools/) installed and configured to access the Kubernetes cluster
* A [Persistent Volume Claim (PVC)](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that can be used
  to store datasets and model files.


key: _node_content
string_value:
{"id_": "4e5dc914-578b-4a4d-868a-c894ebf19509", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0a976eb2-9650-4638-be27-0840e80eb0c4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "90bf510721ec54ac392e192edee32d70442722de8c43e41a564ce98d29812cda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7e5c2e21-d17e-4b55-a85d-11fd4bbfe25c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "6abd131e8d06e12a79d46291d1e7f9a32e0551911a560f0875c23652d74a07bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "67c71e14-f6d8-4ebe-a788-b022288c8ab9", "node_type": "1", "metadata": {}, "hash": "91f55f0a24902c6cfe93427e3094539cf298635f5dc3b45065e20bcddd7ac1d6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4936, "end_char_idx": 7955, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```shell script
 cat hostfile
 xxx.xxx.xxx.xxx #node0 ip
 xxx.xxx.xxx.xxx #node1 ip
```
Now, run the following command in node0 and **4DDP** will be enabled in node0 and node1 with BF16 auto mixed precision:
```shell script
 export CCL_WORKER_COUNT=1
 export MASTER_ADDR=xxx.xxx.xxx.xxx #node0 ip
 mpirun -f hostfile -n 4 -ppn 2 \
 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path google-bert/bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex \
 --bf16
```

## Usage with Kubernetes

The same distributed training job from the previous section can be deployed to a Kubernetes cluster using the
[Kubeflow PyTorchJob training operator](https://www.kubeflow.org/docs/components/training/pytorch/).

### Setup

This example assumes that you have:
* Access to a Kubernetes cluster with [Kubeflow installed](https://www.kubeflow.org/docs/started/installing-kubeflow/)
* [`kubectl`](https://kubernetes.io/docs/tasks/tools/) installed and configured to access the Kubernetes cluster
* A [Persistent Volume Claim (PVC)](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that can be used
  to store datasets and model files. There are multiple options for setting up the PVC including using an NFS
  [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) or a cloud storage bucket.
* A Docker container that includes your model training script and all the dependencies needed to run the script. For
  distributed CPU training jobs, this typically includes PyTorch, Transformers, Intel Extension for PyTorch, Intel
  oneCCL Bindings for PyTorch, and OpenSSH to communicate between the containers.

The snippet below is an example of a Dockerfile that uses a base image that supports distributed CPU training and then
extracts a Transformers release to the `/workspace` directory, so that the example scripts are included in the image:
```dockerfile
FROM intel/intel-optimized-pytorch:2.3.0-pip-multinode

RUN apt-get update -y && \
    apt-get install -y --no-install-recommends --fix-missing \
    google-perftools \
    libomp-dev

WORKDIR /workspace

# Download and extract the transformers code
ARG HF_TRANSFORMERS_VER="4.44.0"
RUN pip install --no-cache-dir \
    transformers==${HF_TRANSFORMERS_VER} && \
    mkdir transformers && \
    curl -sSL --retry 5 https://github.com/huggingface/transformers/archive/refs/tags/v${HF_TRANSFORMERS_VER}.tar.gz | tar -C transformers --strip-components=1 -xzf -
```
The image needs to be built and copied to the cluster's nodes or pushed to a container registry prior to deploying the
PyTorchJob to the cluster.

### PyTorchJob Specification File

The [Kubeflow PyTorchJob](https://www.kubeflow.org/docs/components/training/pytorch/) is used to run the distributed
training job on the cluster.


key: _node_content
string_value:
{"id_": "67c71e14-f6d8-4ebe-a788-b022288c8ab9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0a976eb2-9650-4638-be27-0840e80eb0c4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "90bf510721ec54ac392e192edee32d70442722de8c43e41a564ce98d29812cda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4e5dc914-578b-4a4d-868a-c894ebf19509", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "f282682efa4e18efe3b23c8b97aa85e717cd31609cf6ae7b232cfe2e412255ff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8780f8a1-0d20-4315-9872-01e7e8e01915", "node_type": "1", "metadata": {}, "hash": "3709aaeca6664d421173c5be7b3e25b8ef305d519fb011cf775ee66345c3c80b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6321, "end_char_idx": 8772, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
There are multiple options for setting up the PVC including using an NFS
  [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) or a cloud storage bucket.
* A Docker container that includes your model training script and all the dependencies needed to run the script. For
  distributed CPU training jobs, this typically includes PyTorch, Transformers, Intel Extension for PyTorch, Intel
  oneCCL Bindings for PyTorch, and OpenSSH to communicate between the containers.

The snippet below is an example of a Dockerfile that uses a base image that supports distributed CPU training and then
extracts a Transformers release to the `/workspace` directory, so that the example scripts are included in the image:
```dockerfile
FROM intel/intel-optimized-pytorch:2.3.0-pip-multinode

RUN apt-get update -y && \
    apt-get install -y --no-install-recommends --fix-missing \
    google-perftools \
    libomp-dev

WORKDIR /workspace

# Download and extract the transformers code
ARG HF_TRANSFORMERS_VER="4.44.0"
RUN pip install --no-cache-dir \
    transformers==${HF_TRANSFORMERS_VER} && \
    mkdir transformers && \
    curl -sSL --retry 5 https://github.com/huggingface/transformers/archive/refs/tags/v${HF_TRANSFORMERS_VER}.tar.gz | tar -C transformers --strip-components=1 -xzf -
```
The image needs to be built and copied to the cluster's nodes or pushed to a container registry prior to deploying the
PyTorchJob to the cluster.

### PyTorchJob Specification File

The [Kubeflow PyTorchJob](https://www.kubeflow.org/docs/components/training/pytorch/) is used to run the distributed
training job on the cluster. The yaml file for the PyTorchJob defines parameters such as:
 * The name of the PyTorchJob
 * The number of replicas (workers)
 * The python script and it's parameters that will be used to run the training job
 * The types of resources (node selector, memory, and CPU) needed for each worker
 * The image/tag for the Docker container to use
 * Environment variables
 * A volume mount for the PVC

The volume mount defines a path where the PVC will be mounted in the container for each worker pod. This location can be
used for the dataset, checkpoint files, and the saved model after training completes.

The snippet below is an example of a yaml file for a PyTorchJob with 4 workers running the
[question-answering example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering).


key: _node_content
string_value:
{"id_": "8780f8a1-0d20-4315-9872-01e7e8e01915", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0a976eb2-9650-4638-be27-0840e80eb0c4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "90bf510721ec54ac392e192edee32d70442722de8c43e41a564ce98d29812cda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "67c71e14-f6d8-4ebe-a788-b022288c8ab9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "054b3a0a061118addeadff1caf8a002ca3383d9a70aba7bcc32961d0a56aa9a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ee5e36d-46ea-4697-b79e-27537d50ce33", "node_type": "1", "metadata": {}, "hash": "3a828dc821d444a53b87001c10311bad0a1a5748a89fc0144c9c46c4ac7b285e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7775, "end_char_idx": 11907, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### PyTorchJob Specification File

The [Kubeflow PyTorchJob](https://www.kubeflow.org/docs/components/training/pytorch/) is used to run the distributed
training job on the cluster. The yaml file for the PyTorchJob defines parameters such as:
 * The name of the PyTorchJob
 * The number of replicas (workers)
 * The python script and it's parameters that will be used to run the training job
 * The types of resources (node selector, memory, and CPU) needed for each worker
 * The image/tag for the Docker container to use
 * Environment variables
 * A volume mount for the PVC

The volume mount defines a path where the PVC will be mounted in the container for each worker pod. This location can be
used for the dataset, checkpoint files, and the saved model after training completes.

The snippet below is an example of a yaml file for a PyTorchJob with 4 workers running the
[question-answering example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering).
```yaml
apiVersion: "kubeflow.org/v1"
kind: PyTorchJob
metadata:
  name: transformers-pytorchjob
spec:
  elasticPolicy:
    rdzvBackend: c10d
    minReplicas: 1
    maxReplicas: 4
    maxRestarts: 10
  pytorchReplicaSpecs:
    Worker:
      replicas: 4  # The number of worker pods
      restartPolicy: OnFailure
      template:
        spec:
          containers:
            - name: pytorch
              image: <image name>:<tag>  # Specify the docker image to use for the worker pods
              imagePullPolicy: IfNotPresent
              command: ["/bin/bash", "-c"]
              args:
                - >-
                  cd /workspace/transformers;
                  pip install -r /workspace/transformers/examples/pytorch/question-answering/requirements.txt;
                  source /usr/local/lib/python3.10/dist-packages/oneccl_bindings_for_pytorch/env/setvars.sh;
                  torchrun /workspace/transformers/examples/pytorch/question-answering/run_qa.py \
                    --model_name_or_path distilbert/distilbert-base-uncased \
                    --dataset_name squad \
                    --do_train \
                    --do_eval \
                    --per_device_train_batch_size 12 \
                    --learning_rate 3e-5 \
                    --num_train_epochs 2 \
                    --max_seq_length 384 \
                    --doc_stride 128 \
                    --output_dir /tmp/pvc-mount/output_$(date +%Y%m%d_%H%M%S) \
                    --no_cuda \
                    --ddp_backend ccl \
                    --bf16 \
                    --use_ipex;
              env:
              - name: LD_PRELOAD
                value: "/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.9:/usr/local/lib/libiomp5.so"
              - name: TRANSFORMERS_CACHE
                value: "/tmp/pvc-mount/transformers_cache"
              - name: HF_DATASETS_CACHE
                value: "/tmp/pvc-mount/hf_datasets_cache"
              - name: LOGLEVEL
                value: "INFO"
              - name: CCL_WORKER_COUNT
                value: "1"
              - name: OMP_NUM_THREADS  # Can be tuned for optimal performance
                value: "240"
              resources:
                limits:
                  cpu: 240  # Update the CPU and memory limit values based on your nodes
                  memory: 128Gi
                requests:
                  cpu: 240  # Update the CPU and memory request values based on your nodes
                  memory: 128Gi
              volumeMounts:
              - name: pvc-volume
                mountPath: /tmp/pvc-mount
              - mountPath: /dev/shm
                name: dshm
          restartPolicy: Never
          nodeSelector:  # Optionally use nodeSelector to match a certain node label for the worker pods
            node-type: gnr
          volumes:
          - name: pvc-volume
            persistentVolumeClaim:
              claimName: transformers-pvc
          - name: dshm
            emptyDir:
              medium: Memory
```
To run this example, update the yaml based on your training script and the nodes in your cluster.


key: _node_content
string_value:
{"id_": "1ee5e36d-46ea-4697-b79e-27537d50ce33", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0a976eb2-9650-4638-be27-0840e80eb0c4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "90bf510721ec54ac392e192edee32d70442722de8c43e41a564ce98d29812cda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8780f8a1-0d20-4315-9872-01e7e8e01915", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3745, "retrieve_doc": true, "source": "transformers"}, "hash": "aa268a565fb3e1940a61c23bb6f5da12e12bdb419aaad3c830542560b94a8e8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11909, "end_char_idx": 14737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

The CPU resource limits/requests in the yaml are defined in [cpu units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu)
where 1 CPU unit is equivalent to 1 physical CPU core or 1 virtual core (depending on whether the node is a physical
host or a VM). The amount of CPU and memory limits/requests defined in the yaml should be less than the amount of
available CPU/memory capacity on a single machine. It is usually a good idea to not use the entire machine's capacity in
order to leave some resources for the kubelet and OS. In order to get ["guaranteed"](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed)
[quality of service](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/) for the worker pods,
set the same CPU and memory amounts for both the resource limits and requests.

</Tip>

### Deploy

After the PyTorchJob spec has been updated with values appropriate for your cluster and training job, it can be deployed
to the cluster using:
```bash
export NAMESPACE=<specify your namespace>

kubectl create -f pytorchjob.yaml -n ${NAMESPACE}
```

The `kubectl get pods -n ${NAMESPACE}` command can then be used to list the pods in your namespace. You should see
the worker pods for the PyTorchJob that was just deployed. At first, they will probably have a status of "Pending" as
the containers get pulled and created, then the status should change to "Running".
```
NAME                                                     READY   STATUS                  RESTARTS          AGE
...
transformers-pytorchjob-worker-0                         1/1     Running                 0                 7m37s
transformers-pytorchjob-worker-1                         1/1     Running                 0                 7m37s
transformers-pytorchjob-worker-2                         1/1     Running                 0                 7m37s
transformers-pytorchjob-worker-3                         1/1     Running                 0                 7m37s
...
```

The logs for worker can be viewed using `kubectl logs <pod name> -n ${NAMESPACE}`. Add `-f` to stream the logs, for example:
```bash
kubectl logs transformers-pytorchjob-worker-0 -n ${NAMESPACE} -f
```

After the training job completes, the trained model can be copied from the PVC or storage location. When you are done
with the job, the PyTorchJob resource can be deleted from the cluster using `kubectl delete -f pytorchjob.yaml -n ${NAMESPACE}`.

## Summary

This guide covered running distributed PyTorch training jobs using multiple CPUs on bare metal and on a Kubernetes
cluster. Both cases utilize Intel Extension for PyTorch and Intel oneCCL Bindings for PyTorch for optimal training
performance, and can be used as a template to run your own workload on multiple nodes.


key: _node_content
string_value:
{"id_": "b574880f-18f4-47cd-b3ad-45061fb60e8a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3429a14-958d-4f31-8c2b-a32ff4a1a597", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "hash": "fcbf39d53d83c6f064cccf34a7654672d76b41edb3c0d340b128169d06afb319", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "18b8e036-a6e7-4d49-8d33-636687d3bf73", "node_type": "1", "metadata": {}, "hash": "e470205c927e63b5257be0726a9e40a2cb53fc05f060c8e73d6a930dc7d75c45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3168, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Padding and truncation

Batched inputs are often different lengths, so they can't be converted to fixed-size tensors. Padding and truncation are strategies for dealing with this problem, to create rectangular tensors from batches of varying lengths. Padding adds a special **padding token** to ensure shorter sequences will have the same length as either the longest sequence in a batch or the maximum length accepted by the model. Truncation works in the other direction by truncating long sequences.

In most cases, padding your batch to the length of the longest sequence and truncating to the maximum length a model can accept works pretty well. However, the API supports more strategies if you need them. The three arguments you need to are: `padding`, `truncation` and `max_length`.

The `padding` argument controls padding. It can be a boolean or a string:

  - `True` or `'longest'`: pad to the longest sequence in the batch (no padding is applied if you only provide
    a single sequence).
  - `'max_length'`: pad to a length specified by the `max_length` argument or the maximum length accepted
    by the model if no `max_length` is provided (`max_length=None`). Padding will still be applied if you only provide a single sequence.
  - `False` or `'do_not_pad'`: no padding is applied. This is the default behavior.

The `truncation` argument controls truncation. It can be a boolean or a string:

  - `True` or `'longest_first'`: truncate to a maximum length specified by the `max_length` argument or
    the maximum length accepted by the model if no `max_length` is provided (`max_length=None`). This will
    truncate token by token, removing a token from the longest sequence in the pair until the proper length is
    reached.
  - `'only_second'`: truncate to a maximum length specified by the `max_length` argument or the maximum
    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate
    the second sentence of a pair if a pair of sequences (or a batch of pairs of sequences) is provided.
  - `'only_first'`: truncate to a maximum length specified by the `max_length` argument or the maximum
    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate
    the first sentence of a pair if a pair of sequences (or a batch of pairs of sequences) is provided.
  - `False` or `'do_not_truncate'`: no truncation is applied. This is the default behavior.

The `max_length` argument controls the length of the padding and truncation. It can be an integer or `None`, in which case it will default to the maximum length the model can accept. If the model has no specific maximum input length, truncation or padding to `max_length` is deactivated.

The following table summarizes the recommended way to setup padding and truncation. If you use pairs of input sequences in any of the following examples, you can replace `truncation=True` by a `STRATEGY` selected in
`['only_first', 'only_second', 'longest_first']`, i.e. `truncation='only_second'` or `truncation='longest_first'` to control how both sequences in the pair are truncated as detailed before.


key: _node_content
string_value:
{"id_": "18b8e036-a6e7-4d49-8d33-636687d3bf73", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3429a14-958d-4f31-8c2b-a32ff4a1a597", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "hash": "fcbf39d53d83c6f064cccf34a7654672d76b41edb3c0d340b128169d06afb319", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b574880f-18f4-47cd-b3ad-45061fb60e8a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "hash": "c63f8575ca6beac956d1366d237eb2a23446ea0293863d38ffdcf7640921b0a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1613, "end_char_idx": 6842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This will
    truncate token by token, removing a token from the longest sequence in the pair until the proper length is
    reached.
  - `'only_second'`: truncate to a maximum length specified by the `max_length` argument or the maximum
    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate
    the second sentence of a pair if a pair of sequences (or a batch of pairs of sequences) is provided.
  - `'only_first'`: truncate to a maximum length specified by the `max_length` argument or the maximum
    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate
    the first sentence of a pair if a pair of sequences (or a batch of pairs of sequences) is provided.
  - `False` or `'do_not_truncate'`: no truncation is applied. This is the default behavior.

The `max_length` argument controls the length of the padding and truncation. It can be an integer or `None`, in which case it will default to the maximum length the model can accept. If the model has no specific maximum input length, truncation or padding to `max_length` is deactivated.

The following table summarizes the recommended way to setup padding and truncation. If you use pairs of input sequences in any of the following examples, you can replace `truncation=True` by a `STRATEGY` selected in
`['only_first', 'only_second', 'longest_first']`, i.e. `truncation='only_second'` or `truncation='longest_first'` to control how both sequences in the pair are truncated as detailed before.

| Truncation                           | Padding                           | Instruction                                                                                 |
|--------------------------------------|-----------------------------------|---------------------------------------------------------------------------------------------|
| no truncation                        | no padding                        | `tokenizer(batch_sentences)`                                                           |
|                                      | padding to max sequence in batch  | `tokenizer(batch_sentences, padding=True)` or                                          |
|                                      |                                   | `tokenizer(batch_sentences, padding='longest')`                                        |
|                                      | padding to max model input length | `tokenizer(batch_sentences, padding='max_length')`                                     |
|                                      | padding to specific length        | `tokenizer(batch_sentences, padding='max_length', max_length=42)`                      |
|                                      | padding to a multiple of a value  | `tokenizer(batch_sentences, padding=True, pad_to_multiple_of=8)`                        |
| truncation to max model input length | no padding                        | `tokenizer(batch_sentences, truncation=True)` or                                       |
|                                      |                                   | `tokenizer(batch_sentences, truncation=STRATEGY)`                                      |
|                                      | padding to max sequence in batch  | `tokenizer(batch_sentences, padding=True, truncation=True)` or                         |
|                                      |                                   | `tokenizer(batch_sentences, padding=True, truncation=STRATEGY)`                        |
|                                      | padding to max model input length | `tokenizer(batch_sentences, padding='max_length', truncation=True)` or                 |
|                                      |                                   | `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY)`                |
|                                      | padding to specific length        | Not possible                                                                                |
| truncation to specific length        | no padding                        | `tokenizer(batch_sentences, truncation=True, max_length=42)` or                        |
|                                      |                                   | `tokenizer(batch_sentences, truncation=STRATEGY, max_length=42)`                       |
|                                      | padding to max sequence in batch  | `tokenizer(batch_sentences, padding=True, truncation=True, max_length=42)` or          |
|                                      |                                   | `tokenizer(batch_sentences, padding=True, truncation=STRATEGY, max_length=42)`         |
|                                      | padding to max model input length | Not possible                                                                                |
|                                      | padding to specific length        | `tokenizer(batch_sentences, padding='max_length', truncation=True, max_length=42)` or  |
|                                      |                                   | `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY, max_length=42)` |


key: _node_content
string_value:
{"id_": "54b25fbf-44ae-44d9-9fce-71f09c126697", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "122ef393-e351-4677-b651-6d54dbe3a0b1", "node_type": "1", "metadata": {}, "hash": "534c67e5180c6deb34ef09f181d5c4b2342c6858616dcd8c6733755a0d246a9e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3948, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Best Practices for Generation with Cache

Efficient caching is crucial for optimizing the performance of models in various generative tasks,
including text generation, translation, summarization and other transformer-based applications.
Effective caching helps reduce computation time and improve response rates, especially in real-time or resource-intensive applications.

Transformers support various caching methods, leveraging "Cache" classes to abstract and manage the caching logic.
This document outlines best practices for using these classes to maximize performance and efficiency.
Check out all the available `Cache` classes in the [API documentation](./internal/generation_utils.md).

## What is Cache and why we should care?

Imagine you’re having a conversation with someone, and instead of remembering what was said previously, you have to start from scratch every time you respond. This would be slow and inefficient, right? In the world of Transformer models, a similar concept applies, and that's where Caching keys and values come into play. From now on, I'll refer to the concept as KV Cache.

KV cache is needed to optimize the generation in autoregressive models, where the model predicts text token by token. This process can be slow since the model can generate only one token at a time, and each new prediction is dependent on the previous context. That means, to predict token number 1000 in the generation, you need information from the previous 999 tokens, which comes in the form of some matrix multiplications across the representations of those tokens. But to predict token number 1001, you also need the same information from the first 999 tokens, plus additional information from token number 1000. That is where key-value cache is used to optimize the sequential generation process by storing previous calculations to reuse in subsequent tokens, so they don't need to be computed again.

More concretely, key-value cache acts as a memory bank for these generative models, where the model stores key-value pairs derived from self-attention layers for previously processed tokens. By storing this information, the model can avoid redundant computations and instead retrieve keys and values of previous tokens from the cache.

<details>
  <summary><em>For the Curious Minds Who Like to Dive Deep</em></summary>

  ### Under the Hood: How Cache Object Works in Attention Mechanism

  When utilizing a cache object in the input, the Attention module performs several critical steps to integrate past and present information seamlessly.

  The Attention module concatenates the current key-values with the past key-values stored in the cache. This results in attention weights of shape `(new_tokens_length, past_kv_length + new_tokens_length)`. Essentially, the past and current key-values are combined to compute attention scores, ensuring that the model considers both previous context and new input. The concatenated key-values are used to compute the attention scores resulting in attention weights of shape `(new_tokens_length, past_kv_length + new_tokens_length)`.

  Therefore, when iteratively calling `forward()` instead of the `generate()` method, it’s crucial to ensure that the attention mask shape matches the combined length of past and current key-values. The attention mask should have the shape `(batch_size, past_kv_length + new_tokens_length)`. This is usually handled internally when you call `generate()` method. If you want to implement your own generation loop with Cache classes, take this into consideration and prepare the attention mask to hold values to current and past tokens.

  <Tip warning={true}>

  One important concept you need to know when writing your own generation loop, is `cache_position`. In case you want to reuse an already filled Cache object by calling `forward()`, you have to pass in a valid `cache_position` which will indicate the positions of inputs in the sequence.


key: _node_content
string_value:
{"id_": "122ef393-e351-4677-b651-6d54dbe3a0b1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "54b25fbf-44ae-44d9-9fce-71f09c126697", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "8be2f00d864f2e3330be7247b09d8c8cb4c44d7aafd6a099644ac11f4bd13d39", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9002990c-3cab-464e-a9dc-924573a9e94d", "node_type": "1", "metadata": {}, "hash": "6b9e19fd4fec58451ecd7f8b9e3040cdca9b71c6b3ef314385d85e0728a5145e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2114, "end_char_idx": 4227, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
By storing this information, the model can avoid redundant computations and instead retrieve keys and values of previous tokens from the cache.

<details>
  <summary><em>For the Curious Minds Who Like to Dive Deep</em></summary>

  ### Under the Hood: How Cache Object Works in Attention Mechanism

  When utilizing a cache object in the input, the Attention module performs several critical steps to integrate past and present information seamlessly.

  The Attention module concatenates the current key-values with the past key-values stored in the cache. This results in attention weights of shape `(new_tokens_length, past_kv_length + new_tokens_length)`. Essentially, the past and current key-values are combined to compute attention scores, ensuring that the model considers both previous context and new input. The concatenated key-values are used to compute the attention scores resulting in attention weights of shape `(new_tokens_length, past_kv_length + new_tokens_length)`.

  Therefore, when iteratively calling `forward()` instead of the `generate()` method, it’s crucial to ensure that the attention mask shape matches the combined length of past and current key-values. The attention mask should have the shape `(batch_size, past_kv_length + new_tokens_length)`. This is usually handled internally when you call `generate()` method. If you want to implement your own generation loop with Cache classes, take this into consideration and prepare the attention mask to hold values to current and past tokens.

  <Tip warning={true}>

  One important concept you need to know when writing your own generation loop, is `cache_position`. In case you want to reuse an already filled Cache object by calling `forward()`, you have to pass in a valid `cache_position` which will indicate the positions of inputs in the sequence. Note that `cache_position` is not affected by padding, and always adds one more position for each token. For example, if key/value cache contains 10 tokens (no matter how many of it is a pad token), the cache position for the next token should be `torch.tensor([10])`.

  </Tip>


key: _node_content
string_value:
{"id_": "9002990c-3cab-464e-a9dc-924573a9e94d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "122ef393-e351-4677-b651-6d54dbe3a0b1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "c5a4416e41be148d32c415dcf40646220a64c0a403f5784c8637d19250f26a77", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "52b82913-23c4-4593-b245-fea22360a132", "node_type": "1", "metadata": {}, "hash": "7d4d5ab9460b0d5100cb8d590f166b3ee092fd7da15d194839a7a6410a574ed5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2569, "end_char_idx": 6117, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Attention module concatenates the current key-values with the past key-values stored in the cache. This results in attention weights of shape `(new_tokens_length, past_kv_length + new_tokens_length)`. Essentially, the past and current key-values are combined to compute attention scores, ensuring that the model considers both previous context and new input. The concatenated key-values are used to compute the attention scores resulting in attention weights of shape `(new_tokens_length, past_kv_length + new_tokens_length)`.

  Therefore, when iteratively calling `forward()` instead of the `generate()` method, it’s crucial to ensure that the attention mask shape matches the combined length of past and current key-values. The attention mask should have the shape `(batch_size, past_kv_length + new_tokens_length)`. This is usually handled internally when you call `generate()` method. If you want to implement your own generation loop with Cache classes, take this into consideration and prepare the attention mask to hold values to current and past tokens.

  <Tip warning={true}>

  One important concept you need to know when writing your own generation loop, is `cache_position`. In case you want to reuse an already filled Cache object by calling `forward()`, you have to pass in a valid `cache_position` which will indicate the positions of inputs in the sequence. Note that `cache_position` is not affected by padding, and always adds one more position for each token. For example, if key/value cache contains 10 tokens (no matter how many of it is a pad token), the cache position for the next token should be `torch.tensor([10])`.

  </Tip>


  See an example below for how to implement your own generation loop.
    
  ```python
  >>> import torch
  >>> from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache
 
  >>> model_id = "meta-llama/Llama-2-7b-chat-hf"
  >>> model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="cuda:0")
  >>> tokenizer = AutoTokenizer.from_pretrained(model_id)

  >>> past_key_values = DynamicCache()
  >>> messages = [{"role": "user", "content": "Hello, what's your name."}]
  >>> inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda:0")

  >>> generated_ids = inputs.input_ids
  >>> cache_position = torch.arange(inputs.input_ids.shape[1], dtype=torch.int64, device="cuda:0")
  >>> max_new_tokens = 10

  >>> for _ in range(max_new_tokens):
  ...     outputs = model(**inputs, cache_position=cache_position, past_key_values=past_key_values, use_cache=True)     
  ...     # Greedily sample one next token
  ...     next_token_ids = outputs.logits[:, -1:].argmax(-1)
  ...     generated_ids = torch.cat([generated_ids, next_token_ids], dim=-1)   
  ...
  ...     # Prepare inputs for the next generation step by leaaving unprocessed tokens, in our case we have only one new token
  ...     # and expanding attn mask for the new token, as explained above
  ...     attention_mask = inputs["attention_mask"]
  ...     attention_mask = torch.cat([attention_mask, attention_mask.new_ones((attention_mask.shape[0], 1))], dim=-1)
  ...     inputs = {"input_ids": next_token_ids, "attention_mask": attention_mask}
  ...     cache_position = cache_position[-1:] + 1 # add one more position for the next token

  >>> print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0])
  "[INST] Hello, what's your name. [/INST]  Hello! My name is LLaMA,"
  ```

</details>


key: _node_content
string_value:
{"id_": "52b82913-23c4-4593-b245-fea22360a132", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9002990c-3cab-464e-a9dc-924573a9e94d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "9e3b991d597d22c2e0e1b54202cc3f63961d1a49d916f721f9f8f5094f120edd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "38430138-73d7-4305-821b-b959870ba674", "node_type": "1", "metadata": {}, "hash": "66ddfb932d517cf286fda87ab14f95d1732cf5f2efa4eb809e5c2844b1bb10fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6121, "end_char_idx": 9664, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Generate with Cache

In 🤗 Transformers, we support various Cache types to optimize the performance across different models and tasks. By default, all models generate with caching,
with the [`~DynamicCache`] class being the default cache for most models. It allows us to dynamically grow cache size, by saving more and more keys and values as we generate. If for some reason you don't want to use caches, you can pass `use_cache=False` into the `generate()` method.

Refer to the table below to see the difference between cache types and choose the one that suits best for your use-case.

| Cache Type          | Memory Efficient | Supports torch.compile() | Initialization Recommended | Latency  |  Long Context Generation |
|---------------------|------------------|--------------------------|----------------------------|----------|--------------------------|
| Dynamic Cache       |      No          |        No                |         No                 |   Mid    |     No                   |
| Static Cache        |      No          |        Yes               |         Yes                |   High   |     No                   |
| Quantized Cache     |      Yes         |        No                |         No                 |   Low    |     Yes                  |
| Offloaded Cache     |      Yes         |        No                |         No                 |   Low    |     No                   |
| Sliding Window Cache|      No          |        Yes               |         Yes                |   High   |     No                   |
| Sink Cache          |      Yes         |        No                |         Yes                |   Mid    |     Yes                  |


These cache classes can be set with a `cache_implementation` argument when generating. To learn about the available options for the cache_implementation flag, please refer to the [API Documentation](./main_classes/text_generation.md#transformers.GenerationConfig). Now, let's explore each cache type in detail and see how to use them. Note that the below examples are for decoder-only Tranformer-based models. We also support ["Model-Specific Cache"] classes for models such as Mamba or Jamba, keep reading for more details.

### Quantized Cache

The key and value cache can occupy a large portion of memory, becoming a [bottleneck for long-context generation](https://huggingface.co/blog/llama31#inference-memory-requirements), especially for Large Language Models.
Quantizing the cache when using `generate()` can significantly reduce memory requirements at the cost of speed.

KV Cache quantization in `transformers` is largely inspired by the paper ["KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache"](https://arxiv.org/abs/2402.02750) and currently supports [`~QuantoQuantizedCache`] and [`~HQQQuantizedCache`] classes. For more information on the inner workings see the paper.

To enable quantization of the key-value cache, one needs to indicate `cache_implementation="quantized"` in the `generation_config`.
Quantization related arguments should be passed to the `generation_config` either as a `dict` or an instance of a [`~QuantizedCacheConfig`] class.
One has to indicate which quantization backend to use in the [`~QuantizedCacheConfig`], the default is `quanto`.

<Tip warning={true}>

Cache quantization can be detrimental in terms of latency if the context length is short and there is enough GPU VRAM available to run without cache quantization. It is recommended to seek balance between memory efficiency and latency.
</Tip>


key: _node_content
string_value:
{"id_": "38430138-73d7-4305-821b-b959870ba674", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "52b82913-23c4-4593-b245-fea22360a132", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "eacbba2668f333eb8e7c8ce5878486c9ff4d25a9832bc51b4aed2676a69f2270", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6a276a67-1308-4779-976d-107290ac366b", "node_type": "1", "metadata": {}, "hash": "66321d2c8b0a79041d2bd1be2c5ad781b7f8cb438580d2ea2b1b3851e9520af1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9667, "end_char_idx": 12713, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
>>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16).to("cuda:0")
>>> inputs = tokenizer("I like rock music because", return_tensors="pt").to(model.device)

>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=20, cache_implementation="quantized", cache_config={"nbits": 4, "backend": "quanto"})
>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
I like rock music because it's loud and energetic. It's a great way to express myself and rel

>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=20)
>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
I like rock music because it's loud and energetic. I like to listen to it when I'm feeling
```

## OffloadedCache

Similarly to KV cache quantization, [`~OffloadedCache`] strategy aims to reduce GPU VRAM usage.
It does so by moving the KV cache for most layers to the CPU.
As the model's `forward()` method iterates over the layers, this strategy maintains the current layer cache on the GPU.
At the same time it asynchronously prefetches the next layer cache as well as sending the previous layer cache back to the CPU.
Unlike KV cache quantization, this strategy always produces the same result as the default KV cache implementation.
Thus, it can serve as a drop-in replacement or a fallback for it.

Depending on your model and the characteristics of your generation task (size of context, number of generated tokens, number of beams, etc.)
you may notice a small degradation in generation throughput compared to the default KV cache implementation.

To enable KV cache offloading, pass `cache_implementation="offloaded"` in the `generation_config` or directky to the `generate()` call.

```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> ckpt = "microsoft/Phi-3-mini-4k-instruct"

>>> tokenizer = AutoTokenizer.from_pretrained(ckpt)
>>> model = AutoModelForCausalLM.from_pretrained(ckpt, torch_dtype=torch.float16).to("cuda:0")
>>> inputs = tokenizer("Fun fact: The shortest", return_tensors="pt").to(model.device)

>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=23, cache_implementation="offloaded")
>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
Fun fact: The shortest war in history was between Britain and Zanzibar on August 27, 1896.

>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=23)
>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
Fun fact: The shortest war in history was between Britain and Zanzibar on August 27, 1896.
```

<Tip warning={true}>

Cache offloading requires a GPU and can be slower than dynamic KV cache. Use it if you are getting CUDA out of memory errors.

</Tip>

The example below shows how KV cache offloading can be used as a fallback strategy.


key: _node_content
string_value:
{"id_": "6a276a67-1308-4779-976d-107290ac366b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "38430138-73d7-4305-821b-b959870ba674", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "764c5fd1ed1a8c4a1169f414ee46d024558e85dcbd66adb6c067926527935107", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "68e0bb14-2e04-4ff3-ae6a-ee333b620f48", "node_type": "1", "metadata": {}, "hash": "9926587d4d0cba90492ee1c448c1c0ed8bbd6f6b0502c1bc59b309d27a048b61", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11028, "end_char_idx": 14083, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Unlike KV cache quantization, this strategy always produces the same result as the default KV cache implementation.
Thus, it can serve as a drop-in replacement or a fallback for it.

Depending on your model and the characteristics of your generation task (size of context, number of generated tokens, number of beams, etc.)
you may notice a small degradation in generation throughput compared to the default KV cache implementation.

To enable KV cache offloading, pass `cache_implementation="offloaded"` in the `generation_config` or directky to the `generate()` call.

```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> ckpt = "microsoft/Phi-3-mini-4k-instruct"

>>> tokenizer = AutoTokenizer.from_pretrained(ckpt)
>>> model = AutoModelForCausalLM.from_pretrained(ckpt, torch_dtype=torch.float16).to("cuda:0")
>>> inputs = tokenizer("Fun fact: The shortest", return_tensors="pt").to(model.device)

>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=23, cache_implementation="offloaded")
>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
Fun fact: The shortest war in history was between Britain and Zanzibar on August 27, 1896.

>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=23)
>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
Fun fact: The shortest war in history was between Britain and Zanzibar on August 27, 1896.
```

<Tip warning={true}>

Cache offloading requires a GPU and can be slower than dynamic KV cache. Use it if you are getting CUDA out of memory errors.

</Tip>

The example below shows how KV cache offloading can be used as a fallback strategy.
```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> def resilient_generate(model, *args, **kwargs):
...     oom = False
...     try:
...         return model.generate(*args, **kwargs)
...     except torch.cuda.OutOfMemoryError as e:
...         print(e)
...         print("retrying with cache_implementation='offloaded'")
...         oom = True
...     if oom:
...         torch.cuda.empty_cache()
...         kwargs["cache_implementation"] = "offloaded"
...         return model.generate(*args, **kwargs)
...
...
>>> ckpt = "microsoft/Phi-3-mini-4k-instruct"
>>> tokenizer = AutoTokenizer.from_pretrained(ckpt)
>>> model = AutoModelForCausalLM.from_pretrained(ckpt, torch_dtype=torch.float16).to("cuda:0")
>>> prompt = ["okay "*1000 + "Fun fact: The most"]
>>> inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
>>> beams = { "num_beams": 40, "num_beam_groups": 40, "num_return_sequences": 40, "diversity_penalty": 1.0, "max_new_tokens": 23, "early_stopping": True, }
>>> out = resilient_generate(model, **inputs, **beams)
>>> responses = tokenizer.batch_decode(out[:,-28:], skip_special_tokens=True)
```

On a GPU with 50 GB of RAM, running this code will print
```
CUDA out of memory. Tried to allocate 4.83 GiB. GPU
retrying with cache_implementation='offloaded'
```
before successfully generating 40 beams.


key: _node_content
string_value:
{"id_": "68e0bb14-2e04-4ff3-ae6a-ee333b620f48", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6a276a67-1308-4779-976d-107290ac366b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "ea389fa8e56325829da630efcb9d2d58d5e0559805335734d39029d9550cef30", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dbf9b9b2-cbfd-4d2b-905f-2f8dc8580ef5", "node_type": "1", "metadata": {}, "hash": "d237030fe8bed58da32504f51953913acaed10a66085cddd7c90cda8871b66ce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12568, "end_char_idx": 15764, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Use it if you are getting CUDA out of memory errors.

</Tip>

The example below shows how KV cache offloading can be used as a fallback strategy.
```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> def resilient_generate(model, *args, **kwargs):
...     oom = False
...     try:
...         return model.generate(*args, **kwargs)
...     except torch.cuda.OutOfMemoryError as e:
...         print(e)
...         print("retrying with cache_implementation='offloaded'")
...         oom = True
...     if oom:
...         torch.cuda.empty_cache()
...         kwargs["cache_implementation"] = "offloaded"
...         return model.generate(*args, **kwargs)
...
...
>>> ckpt = "microsoft/Phi-3-mini-4k-instruct"
>>> tokenizer = AutoTokenizer.from_pretrained(ckpt)
>>> model = AutoModelForCausalLM.from_pretrained(ckpt, torch_dtype=torch.float16).to("cuda:0")
>>> prompt = ["okay "*1000 + "Fun fact: The most"]
>>> inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
>>> beams = { "num_beams": 40, "num_beam_groups": 40, "num_return_sequences": 40, "diversity_penalty": 1.0, "max_new_tokens": 23, "early_stopping": True, }
>>> out = resilient_generate(model, **inputs, **beams)
>>> responses = tokenizer.batch_decode(out[:,-28:], skip_special_tokens=True)
```

On a GPU with 50 GB of RAM, running this code will print
```
CUDA out of memory. Tried to allocate 4.83 GiB. GPU
retrying with cache_implementation='offloaded'
```
before successfully generating 40 beams.



### Static Cache

Since the "DynamicCache" dynamically grows with each generation step, it prevents you from taking advantage of JIT optimizations. The [`~StaticCache`] pre-allocates 
a specific maximum size for the keys and values, allowing you to generate up to the maximum length without having to modify cache size. Check the below usage example.

For more examples with Static Cache and JIT compilation, take a look at [StaticCache & torchcompile](./llm_optims.md#static-kv-cache-and-torchcompile)

```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
>>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16, device_map="auto")
>>> inputs = tokenizer("Hello, my name is", return_tensors="pt").to(model.device)

>>> # simply pass the cache implementation="static"
>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=20, cache_implementation="static")
>>> tokenizer.batch_decode(out, skip_special_tokens=True)[0]
"Hello, my name is [Your Name], and I am a [Your Profession] with [Number of Years] of"
```

### Sliding Window Cache

As the name suggests, this cache type implements a sliding window over previous keys and values, retaining only the last `sliding_window` tokens. It should be used with models like Mistral that support sliding window attention. Additionally, similar to Static Cache, this one is JIT-friendly and can be used with the same compile tecniques as Static Cache.

Note that you can use this cache only for models that support sliding window, e.g. Mistral models.


key: _node_content
string_value:
{"id_": "dbf9b9b2-cbfd-4d2b-905f-2f8dc8580ef5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "68e0bb14-2e04-4ff3-ae6a-ee333b620f48", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "9b6fd0b92487c02b6c5fc4b98373620d8aba00716b12ef9f79ca546d45174a4b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4e694473-b3f3-4b8f-97ba-e19277942fb1", "node_type": "1", "metadata": {}, "hash": "1f5a0e6d56913157177267eee6c9259107bb3667020b612074c13fb2ccea0729", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14087, "end_char_idx": 17422, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Static Cache

Since the "DynamicCache" dynamically grows with each generation step, it prevents you from taking advantage of JIT optimizations. The [`~StaticCache`] pre-allocates 
a specific maximum size for the keys and values, allowing you to generate up to the maximum length without having to modify cache size. Check the below usage example.

For more examples with Static Cache and JIT compilation, take a look at [StaticCache & torchcompile](./llm_optims.md#static-kv-cache-and-torchcompile)

```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
>>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16, device_map="auto")
>>> inputs = tokenizer("Hello, my name is", return_tensors="pt").to(model.device)

>>> # simply pass the cache implementation="static"
>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=20, cache_implementation="static")
>>> tokenizer.batch_decode(out, skip_special_tokens=True)[0]
"Hello, my name is [Your Name], and I am a [Your Profession] with [Number of Years] of"
```

### Sliding Window Cache

As the name suggests, this cache type implements a sliding window over previous keys and values, retaining only the last `sliding_window` tokens. It should be used with models like Mistral that support sliding window attention. Additionally, similar to Static Cache, this one is JIT-friendly and can be used with the same compile tecniques as Static Cache.

Note that you can use this cache only for models that support sliding window, e.g. Mistral models. 


```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, SinkCache

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
>>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", torch_dtype=torch.float16).to("cuda:0")
>>> inputs = tokenizer("Yesterday I was on a rock concert and.", return_tensors="pt").to(model.device)

>>> # can be used by passing in cache implementation
>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=30, cache_implementation="sliding_window")
>>> tokenizer.batch_decode(out, skip_special_tokens=True)[0]
"Yesterday I was on a rock concert and. I was so excited to see my favorite band. I was so excited that I was jumping up and down and screaming. I was so excited that I"
```

### Sink Cache

Sink Cache was introduced in ["Efficient Streaming Language Models with Attention Sinks"](https://arxiv.org/abs/2309.17453). It allows you to generate long sequences of text ("infinite length" according to the paper) without any fine-tuning. That is achieved by smart handling of previous keys and values, specifically it retains a few initial tokens from the sequence, called "sink tokens". This is based on the observation that these initial tokens attract a significant portion of attention scores during the generation process. Tokens that come after "sink tokens" are discarded on a sliding windowed basis, keeping only the latest `window_size` tokens. By keeping these initial tokens as "attention sinks," the model maintains stable performance even when dealing with very long texts, thus discarding most of the previous knowledge.


key: _node_content
string_value:
{"id_": "4e694473-b3f3-4b8f-97ba-e19277942fb1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dbf9b9b2-cbfd-4d2b-905f-2f8dc8580ef5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e925d6a9c35d872d4d6dd70ceaefe19132aa6e3642ec809098b0d1a8482d9baf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d47dcb1e-4a53-491d-ba9a-b7f05b45c835", "node_type": "1", "metadata": {}, "hash": "a4f1a3e3617a329db5cd00660825ab2092fbaef121f1282564ed73edfb82394d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15768, "end_char_idx": 19046, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, SinkCache

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
>>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", torch_dtype=torch.float16).to("cuda:0")
>>> inputs = tokenizer("Yesterday I was on a rock concert and.", return_tensors="pt").to(model.device)

>>> # can be used by passing in cache implementation
>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=30, cache_implementation="sliding_window")
>>> tokenizer.batch_decode(out, skip_special_tokens=True)[0]
"Yesterday I was on a rock concert and. I was so excited to see my favorite band. I was so excited that I was jumping up and down and screaming. I was so excited that I"
```

### Sink Cache

Sink Cache was introduced in ["Efficient Streaming Language Models with Attention Sinks"](https://arxiv.org/abs/2309.17453). It allows you to generate long sequences of text ("infinite length" according to the paper) without any fine-tuning. That is achieved by smart handling of previous keys and values, specifically it retains a few initial tokens from the sequence, called "sink tokens". This is based on the observation that these initial tokens attract a significant portion of attention scores during the generation process. Tokens that come after "sink tokens" are discarded on a sliding windowed basis, keeping only the latest `window_size` tokens. By keeping these initial tokens as "attention sinks," the model maintains stable performance even when dealing with very long texts, thus discarding most of the previous knowledge.

Unlike other cache classes, this one can't be used directly by indicating a `cache_implementation`. You have to initialize the Cache before calling on `generate()` as follows.

```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, SinkCache

>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
>>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16).to("cuda:0")
>>> inputs = tokenizer("This is a long story about unicorns, fairies and magic.", return_tensors="pt").to(model.device)

>>> # get our cache, specify number of sink tokens and window size
>>> # Note that window size already includes sink tokens, so has to be larger
>>> past_key_values = SinkCache(window_length=256, num_sink_tokens=4)
>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=30, past_key_values=past_key_values)
>>> tokenizer.batch_decode(out, skip_special_tokens=True)[0]
"This is a long story about unicorns, fairies and magic. It is a fantasy world where unicorns and fairies live together in harmony. The story follows a young girl named Lily"
```

### Encoder-Decoder Cache

The [`~EncoderDecoderCache`] is a wrapper designed to handle the caching needs of encoder-decoder models. This cache type is specifically built to manage both self-attention and cross-attention caches, ensuring storage and retrieval of past key/values required for these complex models. Cool thing about Encoder-Decoder Cache is that you can set different cache types for the encoder and for the decoder, depending on your use case.


key: _node_content
string_value:
{"id_": "d47dcb1e-4a53-491d-ba9a-b7f05b45c835", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4e694473-b3f3-4b8f-97ba-e19277942fb1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "4b5f523380bfdb361e7c37869d7986ae455c872a99d225cf48e6d476fa0d7e52", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "51c91c6b-a86b-45c2-95b4-c6e97460072d", "node_type": "1", "metadata": {}, "hash": "c0b4afc16781aede16d569f054471650d8f67084c69ad536b8687852a3a016cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17424, "end_char_idx": 20905, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Unlike other cache classes, this one can't be used directly by indicating a `cache_implementation`. You have to initialize the Cache before calling on `generate()` as follows.

```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, SinkCache

>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
>>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16).to("cuda:0")
>>> inputs = tokenizer("This is a long story about unicorns, fairies and magic.", return_tensors="pt").to(model.device)

>>> # get our cache, specify number of sink tokens and window size
>>> # Note that window size already includes sink tokens, so has to be larger
>>> past_key_values = SinkCache(window_length=256, num_sink_tokens=4)
>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=30, past_key_values=past_key_values)
>>> tokenizer.batch_decode(out, skip_special_tokens=True)[0]
"This is a long story about unicorns, fairies and magic. It is a fantasy world where unicorns and fairies live together in harmony. The story follows a young girl named Lily"
```

### Encoder-Decoder Cache

The [`~EncoderDecoderCache`] is a wrapper designed to handle the caching needs of encoder-decoder models. This cache type is specifically built to manage both self-attention and cross-attention caches, ensuring storage and retrieval of past key/values required for these complex models. Cool thing about Encoder-Decoder Cache is that you can set different cache types for the encoder and for the decoder, depending on your use case. Currently this cache is only supported in [Whisper](./model_doc/whisper.md) models but we will be adding more models soon. 

In terms of usage, there is nothing special to be done and calling `generate()` or `forward()` will handle everything for you.


### Model-specific Cache Classes

Some models require storing previous keys, values, or states in a specific way, and the above cache classes cannot be used. For such cases, we have several specialized cache classes that are designed for specific models. These models only accept their own dedicated cache classes and do not support using any other cache types. Some examples include [`~HybridCache`] for [Gemma2](./model_doc/gemma2.md) series models or [`~MambaCache`] for [Mamba](./model_doc/mamba.md) architecture models.


## Iterative Generation with Cache

We have seen how to use each of the cache types when generating. What if you want to use cache in iterative generation setting, for example in applications like chatbots, where interactions involve multiple turns and continuous back-and-forth exchanges. Iterative generation with cache allows these systems to handle ongoing conversations effectively without reprocessing the entire context at each step. But there are some tips that you should know before you start implementing:

The general format when doing iterative generation is as below. First you have to initialize an empty cache of the type you want, and you can start feeding in new prompts iteratively. Keeping track of dialogues history and formatting can be done with chat templates, read more on that in [chat_templating](./chat_templating.md)

In case you are using Sink Cache, you have to crop your inputs to that maximum length because Sink Cache can generate text longer than its maximum window size, but it expects the first input to not exceed the maximum cache length.


key: _node_content
string_value:
{"id_": "51c91c6b-a86b-45c2-95b4-c6e97460072d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d47dcb1e-4a53-491d-ba9a-b7f05b45c835", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "0b2e32cdf24340c109c38f21d3bda3ee750405eefc8da9d39a5e2e4fe5b86eb7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4760620d-db8d-4bf9-9dfb-fa1485589418", "node_type": "1", "metadata": {}, "hash": "dfab3a75a4bee2faa5fa2eb24d82db8141a4d1ddc1ceb2d90878bce52f2a5843", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19047, "end_char_idx": 22739, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Currently this cache is only supported in [Whisper](./model_doc/whisper.md) models but we will be adding more models soon. 

In terms of usage, there is nothing special to be done and calling `generate()` or `forward()` will handle everything for you.


### Model-specific Cache Classes

Some models require storing previous keys, values, or states in a specific way, and the above cache classes cannot be used. For such cases, we have several specialized cache classes that are designed for specific models. These models only accept their own dedicated cache classes and do not support using any other cache types. Some examples include [`~HybridCache`] for [Gemma2](./model_doc/gemma2.md) series models or [`~MambaCache`] for [Mamba](./model_doc/mamba.md) architecture models.


## Iterative Generation with Cache

We have seen how to use each of the cache types when generating. What if you want to use cache in iterative generation setting, for example in applications like chatbots, where interactions involve multiple turns and continuous back-and-forth exchanges. Iterative generation with cache allows these systems to handle ongoing conversations effectively without reprocessing the entire context at each step. But there are some tips that you should know before you start implementing:

The general format when doing iterative generation is as below. First you have to initialize an empty cache of the type you want, and you can start feeding in new prompts iteratively. Keeping track of dialogues history and formatting can be done with chat templates, read more on that in [chat_templating](./chat_templating.md)

In case you are using Sink Cache, you have to crop your inputs to that maximum length because Sink Cache can generate text longer than its maximum window size, but it expects the first input to not exceed the maximum cache length.  


```python
>>> import torch
>>> from transformers import AutoTokenizer,AutoModelForCausalLM
>>> from transformers.cache_utils import (
>>>     DynamicCache,
>>>     SinkCache,
>>>     StaticCache,
>>>     SlidingWindowCache,
>>>     QuantoQuantizedCache,
>>>     QuantizedCacheConfig,
>>> )

>>> model_id = "meta-llama/Llama-2-7b-chat-hf"
>>> model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map='auto')
>>> tokenizer = AutoTokenizer.from_pretrained(model_id)

>>> user_prompts = ["Hello, what's your name?", "Btw, yesterday I was on a rock concert."]

>>> past_key_values = DynamicCache()
>>> max_cache_length = past_key_values.get_max_length()

>>> messages = []
>>> for prompt in user_prompts:
...     messages.append({"role": "user", "content": prompt})
...     inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
...     if isinstance(past_key_values, SinkCache):
...         inputs = {k: v[:, -max_cache_length:] for k, v in inputs.items()}
... 
...     input_length = inputs["input_ids"].shape[1]
...     
...     outputs = model.generate(**inputs, do_sample=False, max_new_tokens=256, past_key_values=past_key_values)
...     completion = tokenizer.decode(outputs[0, input_length: ], skip_special_tokens=True)
...     messages.append({"role": "assistant", "content": completion})

print(messages)
[{'role': 'user', 'content': "Hello, what's your name?"}, {'role': 'assistant', 'content': " Hello! My name is LLaMA, I'm a large language model trained by a team of researcher at Meta AI. 😊"}, {'role': 'user', 'content': 'Btw, yesterday I was on a rock concert.'}, {'role': 'assistant', 'content': ' Oh, cool! That sounds like a lot of fun! 🎉 Did you enjoy the concert? What was the band like? 🤔'}]
```


key: _node_content
string_value:
{"id_": "4760620d-db8d-4bf9-9dfb-fa1485589418", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9ad763d2-1ae8-4e91-a34f-c05972c1a583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "e68b35583673d8d22958554efdb9a7a1dccc9600935a4975537e48d1bda7b56c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "51c91c6b-a86b-45c2-95b4-c6e97460072d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/kv_cache", "title": "Best Practices for Generation with Cache", "tokens": 5292, "retrieve_doc": true, "source": "transformers"}, "hash": "20dccc6f086c7c9461449f3996ddc227a0e41b487810ae4ed0012f90e64e8eae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22742, "end_char_idx": 23042, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


