# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 083305f2-b579-4fab-af4e-5cdbfb33d2ba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"083305f2-b579-4fab-af4e-5cdbfb33d2ba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/create_a_model\", \"title\": \"Create a custom architecture\", \"tokens\": 4699, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e9ad52a7-418f-4f38-80d8-afeca381c91e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/create_a_model\", \"title\": \"Create a custom architecture\", \"tokens\": 4699, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4cf01dee-cd3d-4d97-add9-7e6aab05da6b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/create_a_model\", \"title\": \"Create a custom architecture\", \"tokens\": 4699, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a46ed4f592de4ff7b1e7712b0f999506a52a400975ee09c0dd86b7183fc0cb8b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59898cd1-57a3-42d0-9547-5d1f6cf7f86f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f60fa4c524bd51c61f93430adbef83bfb17dbe28ebd3c4be56967f996eff408f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14898, \"end_char_idx\": 18073, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "doc_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "ref_doc_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "chroma:document": "```python\nfrom transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation\n\nconfig = MaskFormerConfig(backbone=\"resnet50\", use_pretrained_backbone=True, use_timm_backbone=True) # backbone and neck config\nmodel = MaskFormerForInstanceSegmentation(config) # head\n```\n\nSet `use_timm_backbone=True` and `use_pretrained_backbone=False` to load a randomly initialized timm backbone.\n\n```python\nfrom transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation\n\nconfig = MaskFormerConfig(backbone=\"resnet50\", use_pretrained_backbone=False, use_timm_backbone=True) # backbone and neck config\nmodel = MaskFormerForInstanceSegmentation(config) # head\n```\n\nYou could also load the backbone config and use it to create a `TimmBackbone` or pass it to the model config. Timm backbones will load pretrained weights by default. Set `use_pretrained_backbone=False` to load randomly initialized weights.\n\n```python\nfrom transformers import TimmBackboneConfig, TimmBackbone\n\nbackbone_config = TimmBackboneConfig(\"resnet50\", use_pretrained_backbone=False)\n\n# Create a backbone class\nbackbone = TimmBackbone(config=backbone_config)\n\n# Create a model with a timm backbone\nfrom transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation\n\nconfig = MaskFormerConfig(backbone_config=backbone_config)\nmodel = MaskFormerForInstanceSegmentation(config)\n```\n\n## Feature extractor\n\nA feature extractor processes audio inputs. It inherits from the base [`~feature_extraction_utils.FeatureExtractionMixin`] class, and may also inherit from the [`SequenceFeatureExtractor`] class for processing audio inputs.\n\nTo use, create a feature extractor associated with the model you're using. For example, create a default [`Wav2Vec2FeatureExtractor`] if you are using [Wav2Vec2](model_doc/wav2vec2) for audio classification:\n\n```py\n>>> from transformers import Wav2Vec2FeatureExtractor\n\n>>> w2v2_extractor = Wav2Vec2FeatureExtractor()\n>>> print(w2v2_extractor)\nWav2Vec2FeatureExtractor {\n  \"do_normalize\": true,\n  \"feature_extractor_type\": \"Wav2Vec2FeatureExtractor\",\n  \"feature_size\": 1,\n  \"padding_side\": \"right\",\n  \"padding_value\": 0.0,\n  \"return_attention_mask\": false,\n  \"sampling_rate\": 16000\n}\n```\n\n<Tip>\n\nIf you aren't looking for any customization, just use the `from_pretrained` method to load a model's default feature extractor parameters.\n\n</Tip>\n\nModify any of the [`Wav2Vec2FeatureExtractor`] parameters to create your custom feature extractor:\n\n```py\n>>> from transformers import Wav2Vec2FeatureExtractor\n\n>>> w2v2_extractor = Wav2Vec2FeatureExtractor(sampling_rate=8000, do_normalize=False)\n>>> print(w2v2_extractor)\nWav2Vec2FeatureExtractor {\n  \"do_normalize\": false,\n  \"feature_extractor_type\": \"Wav2Vec2FeatureExtractor\",\n  \"feature_size\": 1,\n  \"padding_side\": \"right\",\n  \"padding_value\": 0.0,\n  \"return_attention_mask\": false,\n  \"sampling_rate\": 8000\n}\n```\n\n## Processor\n\nFor models that support multimodal tasks, \ud83e\udd17 Transformers offers a processor class that conveniently wraps processing classes such as a feature extractor and a tokenizer into a single object. For example, let's use the [`Wav2Vec2Processor`] for an automatic speech recognition task (ASR)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 59898cd1-57a3-42d0-9547-5d1f6cf7f86f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"59898cd1-57a3-42d0-9547-5d1f6cf7f86f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/create_a_model\", \"title\": \"Create a custom architecture\", \"tokens\": 4699, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e9ad52a7-418f-4f38-80d8-afeca381c91e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/create_a_model\", \"title\": \"Create a custom architecture\", \"tokens\": 4699, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"083305f2-b579-4fab-af4e-5cdbfb33d2ba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/create_a_model\", \"title\": \"Create a custom architecture\", \"tokens\": 4699, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"70377aacd483deb02318033f778c383c6603aec1d077483ab963cec3fdd37223\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16585, \"end_char_idx\": 19176, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "doc_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "ref_doc_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "chroma:document": "For example, create a default [`Wav2Vec2FeatureExtractor`] if you are using [Wav2Vec2](model_doc/wav2vec2) for audio classification:\n\n```py\n>>> from transformers import Wav2Vec2FeatureExtractor\n\n>>> w2v2_extractor = Wav2Vec2FeatureExtractor()\n>>> print(w2v2_extractor)\nWav2Vec2FeatureExtractor {\n  \"do_normalize\": true,\n  \"feature_extractor_type\": \"Wav2Vec2FeatureExtractor\",\n  \"feature_size\": 1,\n  \"padding_side\": \"right\",\n  \"padding_value\": 0.0,\n  \"return_attention_mask\": false,\n  \"sampling_rate\": 16000\n}\n```\n\n<Tip>\n\nIf you aren't looking for any customization, just use the `from_pretrained` method to load a model's default feature extractor parameters.\n\n</Tip>\n\nModify any of the [`Wav2Vec2FeatureExtractor`] parameters to create your custom feature extractor:\n\n```py\n>>> from transformers import Wav2Vec2FeatureExtractor\n\n>>> w2v2_extractor = Wav2Vec2FeatureExtractor(sampling_rate=8000, do_normalize=False)\n>>> print(w2v2_extractor)\nWav2Vec2FeatureExtractor {\n  \"do_normalize\": false,\n  \"feature_extractor_type\": \"Wav2Vec2FeatureExtractor\",\n  \"feature_size\": 1,\n  \"padding_side\": \"right\",\n  \"padding_value\": 0.0,\n  \"return_attention_mask\": false,\n  \"sampling_rate\": 8000\n}\n```\n\n## Processor\n\nFor models that support multimodal tasks, \ud83e\udd17 Transformers offers a processor class that conveniently wraps processing classes such as a feature extractor and a tokenizer into a single object. For example, let's use the [`Wav2Vec2Processor`] for an automatic speech recognition task (ASR). ASR transcribes audio to text, so you will need a feature extractor and a tokenizer.\n\nCreate a feature extractor to handle the audio inputs:\n\n```py\n>>> from transformers import Wav2Vec2FeatureExtractor\n\n>>> feature_extractor = Wav2Vec2FeatureExtractor(padding_value=1.0, do_normalize=True)\n```\n\nCreate a tokenizer to handle the text inputs:\n\n```py\n>>> from transformers import Wav2Vec2CTCTokenizer\n\n>>> tokenizer = Wav2Vec2CTCTokenizer(vocab_file=\"my_vocab_file.txt\")\n```\n\nCombine the feature extractor and tokenizer in [`Wav2Vec2Processor`]:\n\n```py\n>>> from transformers import Wav2Vec2Processor\n\n>>> processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)\n```\n\nWith two basic classes - configuration and model - and an additional preprocessing class (tokenizer, image processor, feature extractor, or processor), you can create any of the models supported by \ud83e\udd17 Transformers. Each of these base classes are configurable, allowing you to use the specific attributes you want. You can easily setup a model for training or modify an existing pretrained model to fine-tune."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cc12e4c1-adcc-4978-b258-f5ce7216660d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cc12e4c1-adcc-4978-b258-f5ce7216660d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2e853d92-2c11-40d0-81fd-ac5e16e85110\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1d294bb065ef66ccb5a16ce1cead7cfbf88423f3d17eca5286d30ac185b77512\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 4663, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "# Methods and tools for efficient training on a single GPU\n\nThis guide demonstrates practical techniques that you can use to increase the efficiency of your model's training by \noptimizing memory utilization, speeding up the training, or both. If you'd like to understand how GPU is utilized during \ntraining, please refer to the [Model training anatomy](model_memory_anatomy) conceptual guide first. This guide \nfocuses on practical techniques.  \n\n<Tip>\n\nIf you have access to a machine with multiple GPUs, these approaches are still valid, plus you can leverage additional methods outlined in the [multi-GPU section](perf_train_gpu_many).\n\n</Tip>\n\nWhen training large models, there are two aspects that should be considered at the same time: \n\n* Data throughput/training time\n* Model performance\n\nMaximizing the throughput (samples/second) leads to lower training cost. This is generally achieved by utilizing the GPU \nas much as possible and thus filling GPU memory to its limit. If the desired batch size exceeds the limits of the GPU memory, \nthe memory optimization techniques, such as gradient accumulation, can help.\n\nHowever, if the preferred batch size fits into memory, there's no reason to apply memory-optimizing techniques because they can \nslow down the training. Just because one can use a large batch size, does not necessarily mean they should. As part of \nhyperparameter tuning, you should determine which batch size yields the best results and then optimize resources accordingly.\n\nThe methods and tools covered in this guide can be classified based on the effect they have on the training process:\n\n| Method/tool                                                                                                                                             | Improves training speed | Optimizes memory utilization |\n|:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------|:-----------------------------|\n| [Batch size choice](#batch-size-choice)                                                                                                                 | Yes                     | Yes                          |\n| [Gradient accumulation](#gradient-accumulation)                                                                                                         | No                      | Yes                          |\n| [Gradient checkpointing](#gradient-checkpointing)                                                                                                       | No                      | Yes                          |\n| [Mixed precision training](#mixed-precision-training)                                                                                                   | Yes                     | Maybe*                       |\n| [torch_empty_cache_steps](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.torch_empty_cache_steps) | No                      | Yes                          |\n| [Optimizer choice](#optimizer-choice)                                                                                                                   | Yes                     | Yes                          |\n| [Data preloading](#data-preloading)                                                                                                                     | Yes                     | No                           |\n| [DeepSpeed Zero](#deepspeed-zero)                                                                                                                       | No                      | Yes                          |\n| [torch.compile](#using-torchcompile)                                                                                                                    | Yes                     | No                           |\n| [Parameter-Efficient Fine Tuning (PEFT)](#using--peft)                                                                                                  | No                      | Yes                          |\n \n<Tip>\n\n*Note: when using mixed precision with a small model and a large batch size, there will be some memory savings but with a \nlarge model and a small batch size, the memory use will be larger.\n\n</Tip>\n\nYou can combine the above methods to get a cumulative effect. These techniques are available to you whether you are \ntraining your model with [`Trainer`] or writing a pure PyTorch loop, in which case you can [configure these optimizations \nwith \ud83e\udd17 Accelerate](#using--accelerate)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2e853d92-2c11-40d0-81fd-ac5e16e85110
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2e853d92-2c11-40d0-81fd-ac5e16e85110\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cc12e4c1-adcc-4978-b258-f5ce7216660d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"33662ed06fd11420e69357661dacf9342b61d4091938221c6ddf331891986828\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bdec2308-f0f9-4982-8a59-4ac147b14ec7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eb294357966085c238f3a027bb664ba22bb21e2e71a2a0c452ee62cfd9986e40\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1502, \"end_char_idx\": 6318, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "The methods and tools covered in this guide can be classified based on the effect they have on the training process:\n\n| Method/tool                                                                                                                                             | Improves training speed | Optimizes memory utilization |\n|:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------|:-----------------------------|\n| [Batch size choice](#batch-size-choice)                                                                                                                 | Yes                     | Yes                          |\n| [Gradient accumulation](#gradient-accumulation)                                                                                                         | No                      | Yes                          |\n| [Gradient checkpointing](#gradient-checkpointing)                                                                                                       | No                      | Yes                          |\n| [Mixed precision training](#mixed-precision-training)                                                                                                   | Yes                     | Maybe*                       |\n| [torch_empty_cache_steps](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.torch_empty_cache_steps) | No                      | Yes                          |\n| [Optimizer choice](#optimizer-choice)                                                                                                                   | Yes                     | Yes                          |\n| [Data preloading](#data-preloading)                                                                                                                     | Yes                     | No                           |\n| [DeepSpeed Zero](#deepspeed-zero)                                                                                                                       | No                      | Yes                          |\n| [torch.compile](#using-torchcompile)                                                                                                                    | Yes                     | No                           |\n| [Parameter-Efficient Fine Tuning (PEFT)](#using--peft)                                                                                                  | No                      | Yes                          |\n \n<Tip>\n\n*Note: when using mixed precision with a small model and a large batch size, there will be some memory savings but with a \nlarge model and a small batch size, the memory use will be larger.\n\n</Tip>\n\nYou can combine the above methods to get a cumulative effect. These techniques are available to you whether you are \ntraining your model with [`Trainer`] or writing a pure PyTorch loop, in which case you can [configure these optimizations \nwith \ud83e\udd17 Accelerate](#using--accelerate).\n\nIf these methods do not result in sufficient gains, you can explore the following options: \n* [Look into building your own custom Docker container with efficient software prebuilds](#efficient-software-prebuilds)\n* [Consider a model that uses Mixture of Experts (MoE)](#mixture-of-experts)\n* [Convert your model to BetterTransformer to leverage PyTorch native attention](#using-pytorch-native-attention-and-flash-attention)\n\nFinally, if all of the above is still not enough, even after switching to a server-grade GPU like A100, consider moving \nto a multi-GPU setup. All these approaches are still valid in a multi-GPU setup, plus you can leverage additional parallelism \ntechniques outlined in the [multi-GPU section](perf_train_gpu_many). \n\n## Batch size choice\n\nTo achieve optimal performance, start by identifying the appropriate batch size. It is recommended to use batch sizes and \ninput/output neuron counts that are of size 2^N. Often it's a multiple of 8, but it can be \nhigher depending on the hardware being used and the model's dtype.\n\nFor reference, check out NVIDIA's recommendation for [input/output neuron counts](\nhttps://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#input-features) and \n[batch size](https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#batch-size) for \nfully connected layers (which are involved in GEMMs (General Matrix Multiplications)).\n\n[Tensor Core Requirements](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc) \ndefine the multiplier based on the dtype and the hardware."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bdec2308-f0f9-4982-8a59-4ac147b14ec7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bdec2308-f0f9-4982-8a59-4ac147b14ec7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2e853d92-2c11-40d0-81fd-ac5e16e85110\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"63165444d87b03c9ae27ec3bbf51f4a7f2b7c64fb01ac7d6626a2905335cb9c7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6c75271d-758b-4c3e-bdfd-8fadee45d571\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"892334beae99862449fa14745627e2e6fb3411fb67ddeac0469caa24e1c160ba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4665, \"end_char_idx\": 8258, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "If these methods do not result in sufficient gains, you can explore the following options: \n* [Look into building your own custom Docker container with efficient software prebuilds](#efficient-software-prebuilds)\n* [Consider a model that uses Mixture of Experts (MoE)](#mixture-of-experts)\n* [Convert your model to BetterTransformer to leverage PyTorch native attention](#using-pytorch-native-attention-and-flash-attention)\n\nFinally, if all of the above is still not enough, even after switching to a server-grade GPU like A100, consider moving \nto a multi-GPU setup. All these approaches are still valid in a multi-GPU setup, plus you can leverage additional parallelism \ntechniques outlined in the [multi-GPU section](perf_train_gpu_many). \n\n## Batch size choice\n\nTo achieve optimal performance, start by identifying the appropriate batch size. It is recommended to use batch sizes and \ninput/output neuron counts that are of size 2^N. Often it's a multiple of 8, but it can be \nhigher depending on the hardware being used and the model's dtype.\n\nFor reference, check out NVIDIA's recommendation for [input/output neuron counts](\nhttps://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#input-features) and \n[batch size](https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#batch-size) for \nfully connected layers (which are involved in GEMMs (General Matrix Multiplications)).\n\n[Tensor Core Requirements](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc) \ndefine the multiplier based on the dtype and the hardware. For instance, for fp16 data type a multiple of 8 is recommended, unless \nit's an A100 GPU, in which case use multiples of 64.\n\nFor parameters that are small, consider also [Dimension Quantization Effects](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#dim-quantization). \nThis is where tiling happens and the right multiplier can have a significant speedup.\n\n## Gradient Accumulation\n\nThe **gradient accumulation** method aims to calculate gradients in smaller increments instead of computing them for the \nentire batch at once. This approach involves iteratively calculating gradients in smaller batches by performing forward \nand backward passes through the model and accumulating the gradients during the process. Once a sufficient number of \ngradients have been accumulated, the model's optimization step is executed. By employing gradient accumulation, it \nbecomes possible to increase the **effective batch size** beyond the limitations imposed by the GPU's memory capacity. \nHowever, it is important to note that the additional forward and backward passes introduced by gradient accumulation can \nslow down the training process.\n\nYou can enable gradient accumulation by adding the `gradient_accumulation_steps` argument to  [`TrainingArguments`]: \n\n```py\ntraining_args = TrainingArguments(per_device_train_batch_size=1, gradient_accumulation_steps=4, **default_args)\n```\n\nIn the above example, your effective batch size becomes 4. \n\nAlternatively, use \ud83e\udd17 Accelerate to gain full control over the training loop. Find the \ud83e\udd17 Accelerate example \n[further down in this guide](#using--accelerate).\n\nWhile it is advised to max out GPU usage as much as possible, a high number of gradient accumulation steps can \nresult in a more pronounced training slowdown. Consider the following example. Let's say, the `per_device_train_batch_size=4` \nwithout gradient accumulation hits the GPU's limit."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6c75271d-758b-4c3e-bdfd-8fadee45d571
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6c75271d-758b-4c3e-bdfd-8fadee45d571\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bdec2308-f0f9-4982-8a59-4ac147b14ec7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"03bca210cb066a00cee89f818cdc36c6076434d8778cc4552becf9f29b940755\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3868e035-743f-4094-b2d8-be72f032853a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"da9605a755e7e7b3993b8b98eae6d58f29cf8b39ebba6fab809a05668083f7a5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6446, \"end_char_idx\": 9864, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "For parameters that are small, consider also [Dimension Quantization Effects](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#dim-quantization). \nThis is where tiling happens and the right multiplier can have a significant speedup.\n\n## Gradient Accumulation\n\nThe **gradient accumulation** method aims to calculate gradients in smaller increments instead of computing them for the \nentire batch at once. This approach involves iteratively calculating gradients in smaller batches by performing forward \nand backward passes through the model and accumulating the gradients during the process. Once a sufficient number of \ngradients have been accumulated, the model's optimization step is executed. By employing gradient accumulation, it \nbecomes possible to increase the **effective batch size** beyond the limitations imposed by the GPU's memory capacity. \nHowever, it is important to note that the additional forward and backward passes introduced by gradient accumulation can \nslow down the training process.\n\nYou can enable gradient accumulation by adding the `gradient_accumulation_steps` argument to  [`TrainingArguments`]: \n\n```py\ntraining_args = TrainingArguments(per_device_train_batch_size=1, gradient_accumulation_steps=4, **default_args)\n```\n\nIn the above example, your effective batch size becomes 4. \n\nAlternatively, use \ud83e\udd17 Accelerate to gain full control over the training loop. Find the \ud83e\udd17 Accelerate example \n[further down in this guide](#using--accelerate).\n\nWhile it is advised to max out GPU usage as much as possible, a high number of gradient accumulation steps can \nresult in a more pronounced training slowdown. Consider the following example. Let's say, the `per_device_train_batch_size=4` \nwithout gradient accumulation hits the GPU's limit. If you would like to train with batches of size 64, do not set the \n`per_device_train_batch_size` to 1 and `gradient_accumulation_steps` to 64. Instead, keep `per_device_train_batch_size=4` \nand set `gradient_accumulation_steps=16`. This results in the same effective batch size while making better use of \nthe available GPU resources.\n\nFor additional information, please refer to batch size and gradient accumulation benchmarks for [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)\nand [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005033957).\n\n## Gradient Checkpointing\n\nSome large models may still face memory issues even when the batch size is set to 1 and gradient accumulation is used. \nThis is because there are other components that also require memory storage.\n\nSaving all activations from the forward pass in order to compute the gradients during the backward pass can result in \nsignificant memory overhead. The alternative approach of discarding the activations and recalculating them when needed \nduring the backward pass, would introduce a considerable computational overhead and slow down the training process.\n\n**Gradient checkpointing** offers a compromise between these two approaches and saves strategically selected activations \nthroughout the computational graph so only a fraction of the activations need to be re-computed for the gradients. For \nan in-depth explanation of gradient checkpointing, refer to [this great article](https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3868e035-743f-4094-b2d8-be72f032853a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3868e035-743f-4094-b2d8-be72f032853a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6c75271d-758b-4c3e-bdfd-8fadee45d571\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"14ec32b5c955abd6993caa094cb597e650b576c879fb0afb34b1e001426ce89a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2aa3d8ea-2a0e-4930-bed7-54f3cf3c24ef\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"44d8cb7ee24f612184b3c99d5457da56a579a26848c689859c000e11e87d8ad7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8127, \"end_char_idx\": 11768, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "Consider the following example. Let's say, the `per_device_train_batch_size=4` \nwithout gradient accumulation hits the GPU's limit. If you would like to train with batches of size 64, do not set the \n`per_device_train_batch_size` to 1 and `gradient_accumulation_steps` to 64. Instead, keep `per_device_train_batch_size=4` \nand set `gradient_accumulation_steps=16`. This results in the same effective batch size while making better use of \nthe available GPU resources.\n\nFor additional information, please refer to batch size and gradient accumulation benchmarks for [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)\nand [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005033957).\n\n## Gradient Checkpointing\n\nSome large models may still face memory issues even when the batch size is set to 1 and gradient accumulation is used. \nThis is because there are other components that also require memory storage.\n\nSaving all activations from the forward pass in order to compute the gradients during the backward pass can result in \nsignificant memory overhead. The alternative approach of discarding the activations and recalculating them when needed \nduring the backward pass, would introduce a considerable computational overhead and slow down the training process.\n\n**Gradient checkpointing** offers a compromise between these two approaches and saves strategically selected activations \nthroughout the computational graph so only a fraction of the activations need to be re-computed for the gradients. For \nan in-depth explanation of gradient checkpointing, refer to [this great article](https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9).\n\nTo enable gradient checkpointing in the [`Trainer`], pass the corresponding a flag to [`TrainingArguments`]:\n\n```py\ntraining_args = TrainingArguments(\n    per_device_train_batch_size=1, gradient_accumulation_steps=4, gradient_checkpointing=True, **default_args\n)\n```\n\nAlternatively, use \ud83e\udd17 Accelerate - find the \ud83e\udd17 Accelerate example [further in this guide](#using--accelerate). \n\n<Tip>\n\nWhile gradient checkpointing may improve memory efficiency, it slows training by approximately 20%.\n\n</Tip>\n\n## Mixed precision training\n\n**Mixed precision training** is a technique that aims to optimize the computational efficiency of training models by \nutilizing lower-precision numerical formats for certain variables. Traditionally, most models use 32-bit floating point \nprecision (fp32 or float32) to represent and process variables. However, not all variables require this high precision \nlevel to achieve accurate results. By reducing the precision of certain variables to lower numerical formats like 16-bit \nfloating point (fp16 or float16), we can speed up the computations. Because in this approach some computations are performed \nin half-precision, while some are still in full precision, the approach is called mixed precision training.\n\nMost commonly mixed precision training is achieved by using fp16 (float16) data types, however, some GPU architectures \n(such as the Ampere architecture) offer bf16 and tf32 (CUDA internal data type) data types. Check \nout the [NVIDIA Blog](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/) to learn more about \nthe differences between these data types.\n\n### fp16\n\nThe main advantage of mixed precision training comes from saving the activations in half precision (fp16). \nAlthough the gradients are also computed in half precision they are converted back to full precision for the optimization \nstep so no memory is saved here."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2aa3d8ea-2a0e-4930-bed7-54f3cf3c24ef
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2aa3d8ea-2a0e-4930-bed7-54f3cf3c24ef\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3868e035-743f-4094-b2d8-be72f032853a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"76ae7d5f524559e6aa10ed38fe45e29e575a30a7a3346896a5587c9cf0c5b447\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9ea3b385-9014-40a4-b246-00794939066e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4b3721e222332d2bf8efbb5456adc6108af4f418d419433dac15209598569963\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10245, \"end_char_idx\": 13584, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "<Tip>\n\nWhile gradient checkpointing may improve memory efficiency, it slows training by approximately 20%.\n\n</Tip>\n\n## Mixed precision training\n\n**Mixed precision training** is a technique that aims to optimize the computational efficiency of training models by \nutilizing lower-precision numerical formats for certain variables. Traditionally, most models use 32-bit floating point \nprecision (fp32 or float32) to represent and process variables. However, not all variables require this high precision \nlevel to achieve accurate results. By reducing the precision of certain variables to lower numerical formats like 16-bit \nfloating point (fp16 or float16), we can speed up the computations. Because in this approach some computations are performed \nin half-precision, while some are still in full precision, the approach is called mixed precision training.\n\nMost commonly mixed precision training is achieved by using fp16 (float16) data types, however, some GPU architectures \n(such as the Ampere architecture) offer bf16 and tf32 (CUDA internal data type) data types. Check \nout the [NVIDIA Blog](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/) to learn more about \nthe differences between these data types.\n\n### fp16\n\nThe main advantage of mixed precision training comes from saving the activations in half precision (fp16). \nAlthough the gradients are also computed in half precision they are converted back to full precision for the optimization \nstep so no memory is saved here. \nWhile mixed precision training results in faster computations, it can also lead to more GPU memory being utilized, especially for small batch sizes.\nThis is because the model is now present on the GPU in both 16-bit and 32-bit precision (1.5x the original model on the GPU).\n\nTo enable mixed precision training, set the `fp16` flag to `True`:\n\n```py\ntraining_args = TrainingArguments(per_device_train_batch_size=4, fp16=True, **default_args)\n```\n\nIf you prefer to use \ud83e\udd17 Accelerate, find the \ud83e\udd17 Accelerate example [further in this guide](#using--accelerate). \n\n### BF16\n\nIf you have access to an Ampere or newer hardware you can use bf16 for mixed precision training and evaluation. While \nbf16 has a worse precision than fp16, it has a much bigger dynamic range. In fp16 the biggest number you can have \nis `65535` and any number above that will result in an overflow. A bf16 number can be as large as `3.39e+38` (!) which \nis about the same as fp32 - because both have 8-bits used for the numerical range.\n\nYou can enable BF16 in the \ud83e\udd17 Trainer with:\n\n```python\ntraining_args = TrainingArguments(bf16=True, **default_args)\n```\n\n### TF32\n\nThe Ampere hardware uses a magical data type called tf32. It has the same numerical range as fp32 (8-bits), but instead \nof 23 bits precision it has only 10 bits (same as fp16) and uses only 19 bits in total. It's \"magical\" in the sense that \nyou can use the normal fp32 training and/or inference code and by enabling tf32 support you can get up to 3x throughput \nimprovement. All you need to do is to add the following to your code:\n\n```python\nimport torch\ntorch.backends.cuda.matmul.allow_tf32 = True\ntorch.backends.cudnn.allow_tf32 = True\n```\n\nCUDA will automatically switch to using tf32 instead of fp32 where possible, assuming that the used GPU is from the Ampere series."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9ea3b385-9014-40a4-b246-00794939066e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9ea3b385-9014-40a4-b246-00794939066e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2aa3d8ea-2a0e-4930-bed7-54f3cf3c24ef\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0d91525d66274f7f2595a19fb84ecb59add85380e23867b615bb5cbebb756536\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"861b2071-664d-4720-a6c5-95230e2b0f2a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"81275c55185587f37ff37d2e5261d369d5a345a48d0ba8b2a4a92abb5cee2638\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12329, \"end_char_idx\": 15435, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "### BF16\n\nIf you have access to an Ampere or newer hardware you can use bf16 for mixed precision training and evaluation. While \nbf16 has a worse precision than fp16, it has a much bigger dynamic range. In fp16 the biggest number you can have \nis `65535` and any number above that will result in an overflow. A bf16 number can be as large as `3.39e+38` (!) which \nis about the same as fp32 - because both have 8-bits used for the numerical range.\n\nYou can enable BF16 in the \ud83e\udd17 Trainer with:\n\n```python\ntraining_args = TrainingArguments(bf16=True, **default_args)\n```\n\n### TF32\n\nThe Ampere hardware uses a magical data type called tf32. It has the same numerical range as fp32 (8-bits), but instead \nof 23 bits precision it has only 10 bits (same as fp16) and uses only 19 bits in total. It's \"magical\" in the sense that \nyou can use the normal fp32 training and/or inference code and by enabling tf32 support you can get up to 3x throughput \nimprovement. All you need to do is to add the following to your code:\n\n```python\nimport torch\ntorch.backends.cuda.matmul.allow_tf32 = True\ntorch.backends.cudnn.allow_tf32 = True\n```\n\nCUDA will automatically switch to using tf32 instead of fp32 where possible, assuming that the used GPU is from the Ampere series.\n\nAccording to [NVIDIA research](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/), the \nmajority of machine learning training workloads show the same perplexity and convergence with tf32 training as with fp32. \nIf you're already using fp16 or bf16 mixed precision it may help with the throughput as well.\n\nYou can enable this mode in the \ud83e\udd17 Trainer:\n\n```python\nTrainingArguments(tf32=True, **default_args)\n```\n\n<Tip>\n\ntf32 can't be accessed directly via `tensor.to(dtype=torch.tf32)` because it is an internal CUDA data type. You need `torch>=1.7` to use tf32 data types.\n\n</Tip>\n\nFor additional information on tf32 vs other precisions, please refer to the following benchmarks: \n[RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803) and\n[A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189).\n\n## Flash Attention 2\n\nYou can speedup the training throughput by using Flash Attention 2 integration in transformers. Check out the appropriate section in the [single GPU section](./perf_infer_gpu_one#Flash-Attention-2) to learn more about how to load a model with Flash Attention 2 modules. \n\n## Optimizer choice\n\nThe most common optimizer used to train transformer models is Adam or AdamW (Adam with weight decay). Adam achieves \ngood convergence by storing the rolling average of the previous gradients; however, it adds an additional memory \nfootprint of the order of the number of model parameters. To remedy this, you can use an alternative optimizer. \nFor example if you have [NVIDIA/apex](https://github.com/NVIDIA/apex) installed for NVIDIA GPUs, or [ROCmSoftwarePlatform/apex](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs, `adamw_apex_fused` will give you the\nfastest training experience among all supported AdamW optimizers."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 861b2071-664d-4720-a6c5-95230e2b0f2a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"861b2071-664d-4720-a6c5-95230e2b0f2a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9ea3b385-9014-40a4-b246-00794939066e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e641d90c68823367efcc89bd3e2fe44aba65a5b010c6a7f34ade38706c2d5d9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ab1b2358-a588-4cba-baaa-461262318cfe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7ca15e68e49fa56cdac15503edeb2bc9ada2eba13b8c85e1385faa9a67fab7cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13830, \"end_char_idx\": 16652, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "If you're already using fp16 or bf16 mixed precision it may help with the throughput as well.\n\nYou can enable this mode in the \ud83e\udd17 Trainer:\n\n```python\nTrainingArguments(tf32=True, **default_args)\n```\n\n<Tip>\n\ntf32 can't be accessed directly via `tensor.to(dtype=torch.tf32)` because it is an internal CUDA data type. You need `torch>=1.7` to use tf32 data types.\n\n</Tip>\n\nFor additional information on tf32 vs other precisions, please refer to the following benchmarks: \n[RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803) and\n[A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189).\n\n## Flash Attention 2\n\nYou can speedup the training throughput by using Flash Attention 2 integration in transformers. Check out the appropriate section in the [single GPU section](./perf_infer_gpu_one#Flash-Attention-2) to learn more about how to load a model with Flash Attention 2 modules. \n\n## Optimizer choice\n\nThe most common optimizer used to train transformer models is Adam or AdamW (Adam with weight decay). Adam achieves \ngood convergence by storing the rolling average of the previous gradients; however, it adds an additional memory \nfootprint of the order of the number of model parameters. To remedy this, you can use an alternative optimizer. \nFor example if you have [NVIDIA/apex](https://github.com/NVIDIA/apex) installed for NVIDIA GPUs, or [ROCmSoftwarePlatform/apex](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs, `adamw_apex_fused` will give you the\nfastest training experience among all supported AdamW optimizers.\n\n[`Trainer`] integrates a variety of optimizers that can be used out of box: `adamw_hf`, `adamw_torch`, `adamw_torch_fused`, \n`adamw_apex_fused`, `adamw_anyprecision`, `adafactor`, or `adamw_bnb_8bit`. More optimizers can be plugged in via a third-party implementation.\n\nLet's take a closer look at two alternatives to AdamW optimizer:\n1. `adafactor` which is available in [`Trainer`]\n2. `adamw_bnb_8bit` is also available in Trainer, but a third-party integration is provided below for demonstration.\n\nFor comparison, for a 3B-parameter model, like \u201cgoogle-t5/t5-3b\u201d: \n* A standard AdamW optimizer will need 24GB of GPU memory because it uses 8 bytes for each parameter (8*3 => 24GB)\n* Adafactor optimizer will need more than 12GB. It uses slightly more than 4 bytes for each parameter, so 4*3 and then some extra.\n* 8bit BNB quantized optimizer will use only (2*3) 6GB if all optimizer states are quantized.\n\n### Adafactor\n\nAdafactor doesn't store rolling averages for each element in weight matrices. Instead, it keeps aggregated information \n(sums of rolling averages row- and column-wise), significantly reducing its footprint. However, compared to Adam, \nAdafactor may have slower convergence in certain cases."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ab1b2358-a588-4cba-baaa-461262318cfe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ab1b2358-a588-4cba-baaa-461262318cfe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"861b2071-664d-4720-a6c5-95230e2b0f2a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c2f23574e66f817044add77084db00decac9de5c09583df22bd1f5cc5dd6c485\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"245247c0-8b05-4aa7-8617-22eeafa0ab2e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2b16bfcf163efe9d449daea34e45248b6aec8a573e35473da48374c8fdff8925\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15145, \"end_char_idx\": 18226, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "For example if you have [NVIDIA/apex](https://github.com/NVIDIA/apex) installed for NVIDIA GPUs, or [ROCmSoftwarePlatform/apex](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs, `adamw_apex_fused` will give you the\nfastest training experience among all supported AdamW optimizers.\n\n[`Trainer`] integrates a variety of optimizers that can be used out of box: `adamw_hf`, `adamw_torch`, `adamw_torch_fused`, \n`adamw_apex_fused`, `adamw_anyprecision`, `adafactor`, or `adamw_bnb_8bit`. More optimizers can be plugged in via a third-party implementation.\n\nLet's take a closer look at two alternatives to AdamW optimizer:\n1. `adafactor` which is available in [`Trainer`]\n2. `adamw_bnb_8bit` is also available in Trainer, but a third-party integration is provided below for demonstration.\n\nFor comparison, for a 3B-parameter model, like \u201cgoogle-t5/t5-3b\u201d: \n* A standard AdamW optimizer will need 24GB of GPU memory because it uses 8 bytes for each parameter (8*3 => 24GB)\n* Adafactor optimizer will need more than 12GB. It uses slightly more than 4 bytes for each parameter, so 4*3 and then some extra.\n* 8bit BNB quantized optimizer will use only (2*3) 6GB if all optimizer states are quantized.\n\n### Adafactor\n\nAdafactor doesn't store rolling averages for each element in weight matrices. Instead, it keeps aggregated information \n(sums of rolling averages row- and column-wise), significantly reducing its footprint. However, compared to Adam, \nAdafactor may have slower convergence in certain cases.\n\nYou can switch to Adafactor by setting `optim=\"adafactor\"` in [`TrainingArguments`]:\n\n```py\ntraining_args = TrainingArguments(per_device_train_batch_size=4, optim=\"adafactor\", **default_args)\n```\n\nCombined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training) \nyou can notice up to 3x improvement while maintaining the throughput! However, as mentioned before, the convergence of \nAdafactor can be worse than Adam. \n\n### 8-bit Adam\n\nInstead of aggregating optimizer states like Adafactor, 8-bit Adam keeps the full state and quantizes it. Quantization \nmeans that it stores the state with lower precision and dequantizes it only for the optimization. This is similar to the \nidea behind mixed precision training.\n\nTo use `adamw_bnb_8bit`, you simply need to set `optim=\"adamw_bnb_8bit\"` in [`TrainingArguments`]:\n\n```py\ntraining_args = TrainingArguments(per_device_train_batch_size=4, optim=\"adamw_bnb_8bit\", **default_args)\n```\n\nHowever, we can also use a third-party implementation of the 8-bit optimizer for demonstration purposes to see how that can be integrated.\n\nFirst, follow the installation guide in the GitHub [repo](https://github.com/TimDettmers/bitsandbytes) to install the `bitsandbytes` library \nthat implements the 8-bit Adam optimizer.\n\nNext you need to initialize the optimizer. This involves two steps: \n* First, group the model's parameters into two groups - one where weight decay should be applied, and the other one where it should not. Usually, biases and layer norm parameters are not weight decayed."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 245247c0-8b05-4aa7-8617-22eeafa0ab2e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"245247c0-8b05-4aa7-8617-22eeafa0ab2e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ab1b2358-a588-4cba-baaa-461262318cfe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0de4b966e5ea07a4c6f7b2c193a2e4f31c73ee961f8f264212076c31c2d88f66\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8e1a536f-145e-4d06-b222-091472956e31\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b6819d4428a3491e37fd7e89bffdfcedb3ceda16e2c099ab3eb2d29af1aa8062\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16569, \"end_char_idx\": 19786, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "However, compared to Adam, \nAdafactor may have slower convergence in certain cases.\n\nYou can switch to Adafactor by setting `optim=\"adafactor\"` in [`TrainingArguments`]:\n\n```py\ntraining_args = TrainingArguments(per_device_train_batch_size=4, optim=\"adafactor\", **default_args)\n```\n\nCombined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training) \nyou can notice up to 3x improvement while maintaining the throughput! However, as mentioned before, the convergence of \nAdafactor can be worse than Adam. \n\n### 8-bit Adam\n\nInstead of aggregating optimizer states like Adafactor, 8-bit Adam keeps the full state and quantizes it. Quantization \nmeans that it stores the state with lower precision and dequantizes it only for the optimization. This is similar to the \nidea behind mixed precision training.\n\nTo use `adamw_bnb_8bit`, you simply need to set `optim=\"adamw_bnb_8bit\"` in [`TrainingArguments`]:\n\n```py\ntraining_args = TrainingArguments(per_device_train_batch_size=4, optim=\"adamw_bnb_8bit\", **default_args)\n```\n\nHowever, we can also use a third-party implementation of the 8-bit optimizer for demonstration purposes to see how that can be integrated.\n\nFirst, follow the installation guide in the GitHub [repo](https://github.com/TimDettmers/bitsandbytes) to install the `bitsandbytes` library \nthat implements the 8-bit Adam optimizer.\n\nNext you need to initialize the optimizer. This involves two steps: \n* First, group the model's parameters into two groups - one where weight decay should be applied, and the other one where it should not. Usually, biases and layer norm parameters are not weight decayed. \n* Then do some argument housekeeping to use the same parameters as the previously used AdamW optimizer.\n\n```py\nimport bitsandbytes as bnb\nfrom torch import nn\nfrom transformers.trainer_pt_utils import get_parameter_names\n\ntraining_args = TrainingArguments(per_device_train_batch_size=4, **default_args)\n\ndecay_parameters = get_parameter_names(model, [nn.LayerNorm])\ndecay_parameters = [name for name in decay_parameters if \"bias\" not in name]\noptimizer_grouped_parameters = [\n    {\n        \"params\": [p for n, p in model.named_parameters() if n in decay_parameters],\n        \"weight_decay\": training_args.weight_decay,\n    },\n    {\n        \"params\": [p for n, p in model.named_parameters() if n not in decay_parameters],\n        \"weight_decay\": 0.0,\n    },\n]\n\noptimizer_kwargs = {\n    \"betas\": (training_args.adam_beta1, training_args.adam_beta2),\n    \"eps\": training_args.adam_epsilon,\n}\noptimizer_kwargs[\"lr\"] = training_args.learning_rate\nadam_bnb_optim = bnb.optim.Adam8bit(\n    optimizer_grouped_parameters,\n    betas=(training_args.adam_beta1, training_args.adam_beta2),\n    eps=training_args.adam_epsilon,\n    lr=training_args.learning_rate,\n)\n```\n\nFinally, pass the custom optimizer as an argument to the `Trainer`:\n\n```py\ntrainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))\n```\n\nCombined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training), \nyou can expect to get about a 3x memory improvement and even slightly higher throughput as using Adafactor."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8e1a536f-145e-4d06-b222-091472956e31
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8e1a536f-145e-4d06-b222-091472956e31\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"245247c0-8b05-4aa7-8617-22eeafa0ab2e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"867185769a723146e497c74818a853319704e6b0b6fb8de9956f8efebd9330f4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f20e9e76-94a5-49d1-b4db-b07e65a62ea3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d2c34d9f7fc4235098958035c9c54c0fd03d75043ca2f9e6a6c3eccbd3d34ad4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18161, \"end_char_idx\": 21625, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "Usually, biases and layer norm parameters are not weight decayed. \n* Then do some argument housekeeping to use the same parameters as the previously used AdamW optimizer.\n\n```py\nimport bitsandbytes as bnb\nfrom torch import nn\nfrom transformers.trainer_pt_utils import get_parameter_names\n\ntraining_args = TrainingArguments(per_device_train_batch_size=4, **default_args)\n\ndecay_parameters = get_parameter_names(model, [nn.LayerNorm])\ndecay_parameters = [name for name in decay_parameters if \"bias\" not in name]\noptimizer_grouped_parameters = [\n    {\n        \"params\": [p for n, p in model.named_parameters() if n in decay_parameters],\n        \"weight_decay\": training_args.weight_decay,\n    },\n    {\n        \"params\": [p for n, p in model.named_parameters() if n not in decay_parameters],\n        \"weight_decay\": 0.0,\n    },\n]\n\noptimizer_kwargs = {\n    \"betas\": (training_args.adam_beta1, training_args.adam_beta2),\n    \"eps\": training_args.adam_epsilon,\n}\noptimizer_kwargs[\"lr\"] = training_args.learning_rate\nadam_bnb_optim = bnb.optim.Adam8bit(\n    optimizer_grouped_parameters,\n    betas=(training_args.adam_beta1, training_args.adam_beta2),\n    eps=training_args.adam_epsilon,\n    lr=training_args.learning_rate,\n)\n```\n\nFinally, pass the custom optimizer as an argument to the `Trainer`:\n\n```py\ntrainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))\n```\n\nCombined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training), \nyou can expect to get about a 3x memory improvement and even slightly higher throughput as using Adafactor. \n\n### multi_tensor\n\npytorch-nightly introduced `torch.optim._multi_tensor` which should significantly speed up the optimizers for situations \nwith lots of small feature tensors. It should eventually become the default, but if you want to experiment with it sooner, take a look at this GitHub [issue](https://github.com/huggingface/transformers/issues/9965).\n\n## Data preloading\n\nOne of the important requirements to reach great training speed is the ability to feed the GPU at the maximum speed it \ncan handle. By default, everything happens in the main process, and it might not be able to read the data from disk fast \nenough, and thus create a bottleneck, leading to GPU under-utilization. Configure the following arguments to reduce the bottleneck:\n\n- `DataLoader(pin_memory=True, ...)` - ensures the data gets preloaded into the pinned memory on CPU and typically leads to much faster transfers from CPU to GPU memory.\n- `DataLoader(num_workers=4, ...)` - spawn several workers to preload data faster. During training, watch the GPU utilization stats; if it's far from 100%, experiment with increasing the number of workers. Of course, the problem could be elsewhere, so many workers won't necessarily lead to better performance.\n\nWhen using [`Trainer`], the corresponding [`TrainingArguments`] are: `dataloader_pin_memory` (`True` by default), and `dataloader_num_workers` (defaults to `0`).\n\n## DeepSpeed ZeRO\n\nDeepSpeed is an open-source deep learning optimization library that is integrated with \ud83e\udd17 Transformers and \ud83e\udd17 Accelerate.\nIt provides a wide range of features and optimizations designed to improve the efficiency and scalability of large-scale \ndeep learning training.\n\nIf your model fits onto a single GPU and you have enough space to fit a small batch size, you don't need to use DeepSpeed\nas it'll only slow things down."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f20e9e76-94a5-49d1-b4db-b07e65a62ea3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f20e9e76-94a5-49d1-b4db-b07e65a62ea3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8e1a536f-145e-4d06-b222-091472956e31\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0795a2b7a461aef9efd6aa0f189ad4b1b7210a613bcd7cc881f5f75cd427391e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"57c275b6-18bb-4500-8c27-3b2745fd7799\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea54760135da62d0f528915ec57b120139cd3e480751c7e3ad3f1c4d67ce63ee\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19789, \"end_char_idx\": 23266, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "### multi_tensor\n\npytorch-nightly introduced `torch.optim._multi_tensor` which should significantly speed up the optimizers for situations \nwith lots of small feature tensors. It should eventually become the default, but if you want to experiment with it sooner, take a look at this GitHub [issue](https://github.com/huggingface/transformers/issues/9965).\n\n## Data preloading\n\nOne of the important requirements to reach great training speed is the ability to feed the GPU at the maximum speed it \ncan handle. By default, everything happens in the main process, and it might not be able to read the data from disk fast \nenough, and thus create a bottleneck, leading to GPU under-utilization. Configure the following arguments to reduce the bottleneck:\n\n- `DataLoader(pin_memory=True, ...)` - ensures the data gets preloaded into the pinned memory on CPU and typically leads to much faster transfers from CPU to GPU memory.\n- `DataLoader(num_workers=4, ...)` - spawn several workers to preload data faster. During training, watch the GPU utilization stats; if it's far from 100%, experiment with increasing the number of workers. Of course, the problem could be elsewhere, so many workers won't necessarily lead to better performance.\n\nWhen using [`Trainer`], the corresponding [`TrainingArguments`] are: `dataloader_pin_memory` (`True` by default), and `dataloader_num_workers` (defaults to `0`).\n\n## DeepSpeed ZeRO\n\nDeepSpeed is an open-source deep learning optimization library that is integrated with \ud83e\udd17 Transformers and \ud83e\udd17 Accelerate.\nIt provides a wide range of features and optimizations designed to improve the efficiency and scalability of large-scale \ndeep learning training.\n\nIf your model fits onto a single GPU and you have enough space to fit a small batch size, you don't need to use DeepSpeed\nas it'll only slow things down. However, if the model doesn't fit onto a single GPU or you can't fit a small batch, you can \nleverage DeepSpeed ZeRO + CPU Offload, or NVMe Offload for much larger models. In this case, you need to separately\n[install the library](main_classes/deepspeed#installation), then follow one of the guides to create a configuration file \nand launch DeepSpeed: \n \n* For an in-depth guide on DeepSpeed integration with [`Trainer`], review [the corresponding documentation](main_classes/deepspeed), specifically the \n[section for a single GPU](main_classes/deepspeed#deployment-with-one-gpu). Some adjustments are required to use DeepSpeed in a notebook; please take a look at the [corresponding guide](main_classes/deepspeed#deployment-in-notebooks).\n* If you prefer to use \ud83e\udd17 Accelerate, refer to [\ud83e\udd17 Accelerate DeepSpeed guide](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed).\n\n## Using torch.compile\n\nPyTorch 2.0 introduced a new compile function that doesn't require any modification to existing PyTorch code but can \noptimize your code by adding a single line of code: `model = torch.compile(model)`.\n\nIf using [`Trainer`], you only need `to` pass the `torch_compile` option in the [`TrainingArguments`]: \n\n```python\ntraining_args = TrainingArguments(torch_compile=True, **default_args)\n```\n\n`torch.compile` uses Python's frame evaluation API to automatically create a graph from existing PyTorch programs. After \ncapturing the graph, different backends can be deployed to lower the graph to an optimized engine. \nYou can find more details and benchmarks in [PyTorch documentation](https://pytorch.org/get-started/pytorch-2.0/)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 57c275b6-18bb-4500-8c27-3b2745fd7799
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"57c275b6-18bb-4500-8c27-3b2745fd7799\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f20e9e76-94a5-49d1-b4db-b07e65a62ea3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8bef8b7c7f10c387163f5db918068b2175903a1d23eee374c1a50e218808e586\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0ec42b3e-cbca-4e76-8753-a7bff3211124\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"60e50f119917ab4c312c0b4a8936ac35b4b376f68ea1811fd22322a4238cfe77\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21626, \"end_char_idx\": 24671, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "However, if the model doesn't fit onto a single GPU or you can't fit a small batch, you can \nleverage DeepSpeed ZeRO + CPU Offload, or NVMe Offload for much larger models. In this case, you need to separately\n[install the library](main_classes/deepspeed#installation), then follow one of the guides to create a configuration file \nand launch DeepSpeed: \n \n* For an in-depth guide on DeepSpeed integration with [`Trainer`], review [the corresponding documentation](main_classes/deepspeed), specifically the \n[section for a single GPU](main_classes/deepspeed#deployment-with-one-gpu). Some adjustments are required to use DeepSpeed in a notebook; please take a look at the [corresponding guide](main_classes/deepspeed#deployment-in-notebooks).\n* If you prefer to use \ud83e\udd17 Accelerate, refer to [\ud83e\udd17 Accelerate DeepSpeed guide](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed).\n\n## Using torch.compile\n\nPyTorch 2.0 introduced a new compile function that doesn't require any modification to existing PyTorch code but can \noptimize your code by adding a single line of code: `model = torch.compile(model)`.\n\nIf using [`Trainer`], you only need `to` pass the `torch_compile` option in the [`TrainingArguments`]: \n\n```python\ntraining_args = TrainingArguments(torch_compile=True, **default_args)\n```\n\n`torch.compile` uses Python's frame evaluation API to automatically create a graph from existing PyTorch programs. After \ncapturing the graph, different backends can be deployed to lower the graph to an optimized engine. \nYou can find more details and benchmarks in [PyTorch documentation](https://pytorch.org/get-started/pytorch-2.0/).\n\n`torch.compile` has a growing list of backends, which can be found in by calling `torchdynamo.list_backends()`, each of which with its optional dependencies.\n\nChoose which backend to use by specifying it via `torch_compile_backend` in the [`TrainingArguments`].  Some of the most commonly used backends are:\n\n**Debugging backends**:\n* `dynamo.optimize(\"eager\")` - Uses PyTorch to run the extracted GraphModule. This is quite useful in debugging TorchDynamo issues.\n* `dynamo.optimize(\"aot_eager\")` - Uses AotAutograd with no compiler, i.e, just using PyTorch eager for the AotAutograd's extracted forward and backward graphs. This is useful for debugging, and unlikely to give speedups.\n\n**Training & inference backends**:\n* `dynamo.optimize(\"inductor\")` - Uses TorchInductor backend with AotAutograd and cudagraphs by leveraging codegened Triton kernels  [Read more](https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747)\n* `dynamo.optimize(\"nvfuser\")` -  nvFuser with TorchScript. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)\n* `dynamo.optimize(\"aot_nvfuser\")` -  nvFuser with AotAutograd. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)\n* `dynamo.optimize(\"aot_cudagraphs\")` - cudagraphs with AotAutograd."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0ec42b3e-cbca-4e76-8753-a7bff3211124
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0ec42b3e-cbca-4e76-8753-a7bff3211124\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"57c275b6-18bb-4500-8c27-3b2745fd7799\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"20f4a91b07fc6961f2f362b506ea632fed596aa20940a048a35ac6ddc44e84f2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"20560969-0b9c-41c5-bc40-6ada9ed8d18e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"29337587d7fc986bff5cb8345aaca667658278785ffe84c1e075a328b736fd5d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23268, \"end_char_idx\": 26002, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "`torch.compile` has a growing list of backends, which can be found in by calling `torchdynamo.list_backends()`, each of which with its optional dependencies.\n\nChoose which backend to use by specifying it via `torch_compile_backend` in the [`TrainingArguments`].  Some of the most commonly used backends are:\n\n**Debugging backends**:\n* `dynamo.optimize(\"eager\")` - Uses PyTorch to run the extracted GraphModule. This is quite useful in debugging TorchDynamo issues.\n* `dynamo.optimize(\"aot_eager\")` - Uses AotAutograd with no compiler, i.e, just using PyTorch eager for the AotAutograd's extracted forward and backward graphs. This is useful for debugging, and unlikely to give speedups.\n\n**Training & inference backends**:\n* `dynamo.optimize(\"inductor\")` - Uses TorchInductor backend with AotAutograd and cudagraphs by leveraging codegened Triton kernels  [Read more](https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747)\n* `dynamo.optimize(\"nvfuser\")` -  nvFuser with TorchScript. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)\n* `dynamo.optimize(\"aot_nvfuser\")` -  nvFuser with AotAutograd. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)\n* `dynamo.optimize(\"aot_cudagraphs\")` - cudagraphs with AotAutograd. [Read more](https://github.com/pytorch/torchdynamo/pull/757)\n\n**Inference-only backend**s:\n* `dynamo.optimize(\"ofi\")` -  Uses Torchscript optimize_for_inference.  [Read more](https://pytorch.org/docs/stable/generated/torch.jit.optimize_for_inference.html)\n* `dynamo.optimize(\"fx2trt\")` -  Uses NVIDIA TensorRT for inference optimizations.  [Read more](https://pytorch.org/TensorRT/tutorials/getting_started_with_fx_path.html)\n* `dynamo.optimize(\"onnxrt\")` -  Uses ONNXRT for inference on CPU/GPU.  [Read more](https://onnxruntime.ai/)\n* `dynamo.optimize(\"ipex\")` -  Uses IPEX for inference on CPU.  [Read more](https://github.com/intel/intel-extension-for-pytorch)\n\nFor an example of using `torch.compile` with \ud83e\udd17 Transformers, check out this [blog post on fine-tuning a BERT model for Text Classification using the newest PyTorch 2.0 features](https://www.philschmid.de/getting-started-pytorch-2-0-transformers)\n\n## Using \ud83e\udd17 PEFT\n\n[Parameter-Efficient Fine Tuning (PEFT)](https://huggingface.co/blog/peft) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it.\n\nAs a result the [memory associated to the optimizer states and gradients](https://huggingface.co/docs/transformers/model_memory_anatomy#anatomy-of-models-memory) are greatly reduced."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 20560969-0b9c-41c5-bc40-6ada9ed8d18e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"20560969-0b9c-41c5-bc40-6ada9ed8d18e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0ec42b3e-cbca-4e76-8753-a7bff3211124\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c339ce2c6460ee94f2972582b759f9187a8ae72ff7713dd1740ed08a51d5cafe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"71b90abc-8637-4c83-9ede-7e087e19476c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a870ec7023092bed4b349be6dea9e3eaa33de491146a52856b3e2a69fe6efb3f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24672, \"end_char_idx\": 27057, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "[Read more](https://github.com/pytorch/torchdynamo/pull/757)\n\n**Inference-only backend**s:\n* `dynamo.optimize(\"ofi\")` -  Uses Torchscript optimize_for_inference.  [Read more](https://pytorch.org/docs/stable/generated/torch.jit.optimize_for_inference.html)\n* `dynamo.optimize(\"fx2trt\")` -  Uses NVIDIA TensorRT for inference optimizations.  [Read more](https://pytorch.org/TensorRT/tutorials/getting_started_with_fx_path.html)\n* `dynamo.optimize(\"onnxrt\")` -  Uses ONNXRT for inference on CPU/GPU.  [Read more](https://onnxruntime.ai/)\n* `dynamo.optimize(\"ipex\")` -  Uses IPEX for inference on CPU.  [Read more](https://github.com/intel/intel-extension-for-pytorch)\n\nFor an example of using `torch.compile` with \ud83e\udd17 Transformers, check out this [blog post on fine-tuning a BERT model for Text Classification using the newest PyTorch 2.0 features](https://www.philschmid.de/getting-started-pytorch-2-0-transformers)\n\n## Using \ud83e\udd17 PEFT\n\n[Parameter-Efficient Fine Tuning (PEFT)](https://huggingface.co/blog/peft) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it.\n\nAs a result the [memory associated to the optimizer states and gradients](https://huggingface.co/docs/transformers/model_memory_anatomy#anatomy-of-models-memory) are greatly reduced.\n\nFor example with a vanilla AdamW, the memory requirement for the optimizer state would be:\n* fp32 copy of parameters: 4 bytes/param\n* Momentum: 4 bytes/param\n* Variance: 4 bytes/param\n\nSuppose a model with 7B parameters and 200 millions parameters injected with [Low Rank Adapters](https://huggingface.co/docs/peft/conceptual_guides/lora).\n\nThe memory requirement for the optimizer state of the plain model would be 12 * 7 = 84 GB (assuming 7B trainable parameters).\n\nAdding Lora increases slightly the memory associated to the model weights and substantially decreases memory requirement for the optimizer state to 12 * 0.2 = 2.4GB.\n\nRead more about PEFT and its detailed usage in [the PEFT documentation](https://huggingface.co/docs/peft/) or [PEFT repository](https://github.com/huggingface/peft).\n\n## Using \ud83e\udd17 Accelerate\n\nWith [\ud83e\udd17 Accelerate](https://huggingface.co/docs/accelerate/index) you can use the above methods while gaining full \ncontrol over the training loop and can essentially write the loop in pure PyTorch with some minor modifications."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 71b90abc-8637-4c83-9ede-7e087e19476c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"71b90abc-8637-4c83-9ede-7e087e19476c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"20560969-0b9c-41c5-bc40-6ada9ed8d18e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d1a10e175b4b787eebec746b077edd7abc855e8d46ec8ac446c0820ec020c3fe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3ba2c6e0-c19a-4c3a-9098-453fd20451e1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ee7f26c7057418433d67e084828b7100097c0bf844c6c1e2ab0e11da48999ed7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25820, \"end_char_idx\": 29183, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "As a result the [memory associated to the optimizer states and gradients](https://huggingface.co/docs/transformers/model_memory_anatomy#anatomy-of-models-memory) are greatly reduced.\n\nFor example with a vanilla AdamW, the memory requirement for the optimizer state would be:\n* fp32 copy of parameters: 4 bytes/param\n* Momentum: 4 bytes/param\n* Variance: 4 bytes/param\n\nSuppose a model with 7B parameters and 200 millions parameters injected with [Low Rank Adapters](https://huggingface.co/docs/peft/conceptual_guides/lora).\n\nThe memory requirement for the optimizer state of the plain model would be 12 * 7 = 84 GB (assuming 7B trainable parameters).\n\nAdding Lora increases slightly the memory associated to the model weights and substantially decreases memory requirement for the optimizer state to 12 * 0.2 = 2.4GB.\n\nRead more about PEFT and its detailed usage in [the PEFT documentation](https://huggingface.co/docs/peft/) or [PEFT repository](https://github.com/huggingface/peft).\n\n## Using \ud83e\udd17 Accelerate\n\nWith [\ud83e\udd17 Accelerate](https://huggingface.co/docs/accelerate/index) you can use the above methods while gaining full \ncontrol over the training loop and can essentially write the loop in pure PyTorch with some minor modifications. \n\nSuppose you have combined the methods in the [`TrainingArguments`] like so:\n\n```py\ntraining_args = TrainingArguments(\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=4,\n    gradient_checkpointing=True,\n    fp16=True,\n    **default_args,\n)\n```\n\nThe full example training loop with \ud83e\udd17 Accelerate is only a handful of lines of code long:\n\n```py\nfrom accelerate import Accelerator\nfrom torch.utils.data.dataloader import DataLoader\n\ndataloader = DataLoader(ds, batch_size=training_args.per_device_train_batch_size)\n\nif training_args.gradient_checkpointing:\n    model.gradient_checkpointing_enable()\n\naccelerator = Accelerator(fp16=training_args.fp16)\nmodel, optimizer, dataloader = accelerator.prepare(model, adam_bnb_optim, dataloader)\n\nmodel.train()\nfor step, batch in enumerate(dataloader, start=1):\n    loss = model(**batch).loss\n    loss = loss / training_args.gradient_accumulation_steps\n    accelerator.backward(loss)\n    if step % training_args.gradient_accumulation_steps == 0:\n        optimizer.step()\n        optimizer.zero_grad()\n```\n\nFirst we wrap the dataset in a [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader). \nThen we can enable gradient checkpointing by calling the model's [`~PreTrainedModel.gradient_checkpointing_enable`] method. \nWhen we initialize the [`Accelerator`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator) \nwe can specify if we want to use mixed precision training and it will take care of it for us in the [`prepare`] call. \nDuring the [`prepare`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator.prepare) \ncall the dataloader will also be distributed across workers should we use multiple GPUs. We use the same [8-bit optimizer](#8-bit-adam) from the earlier example.\n\nFinally, we can add the main training loop. Note that the `backward` call is handled by \ud83e\udd17 Accelerate. We can also see\nhow gradient accumulation works: we normalize the loss, so we get the average at the end of accumulation and once we have \nenough steps we run the optimization."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3ba2c6e0-c19a-4c3a-9098-453fd20451e1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3ba2c6e0-c19a-4c3a-9098-453fd20451e1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"71b90abc-8637-4c83-9ede-7e087e19476c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ed6ad165a90d4c0e6f7c4c0e96a43ccb6c61e25dfd239cf83095f87cdc1d21de\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2050127c-296e-4bd6-8594-50cc2cef38fc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6dd543a9bd13a10453af50206de91af489c86ac8bbb09063ef7506921b74210d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 28240, \"end_char_idx\": 31559, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "Then we can enable gradient checkpointing by calling the model's [`~PreTrainedModel.gradient_checkpointing_enable`] method. \nWhen we initialize the [`Accelerator`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator) \nwe can specify if we want to use mixed precision training and it will take care of it for us in the [`prepare`] call. \nDuring the [`prepare`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator.prepare) \ncall the dataloader will also be distributed across workers should we use multiple GPUs. We use the same [8-bit optimizer](#8-bit-adam) from the earlier example.\n\nFinally, we can add the main training loop. Note that the `backward` call is handled by \ud83e\udd17 Accelerate. We can also see\nhow gradient accumulation works: we normalize the loss, so we get the average at the end of accumulation and once we have \nenough steps we run the optimization. \n\nImplementing these optimization techniques with \ud83e\udd17 Accelerate only takes a handful of lines of code and comes with the \nbenefit of more flexibility in the training loop. For a full documentation of all features have a look at the \n[Accelerate documentation](https://huggingface.co/docs/accelerate/index).\n\n\n## Efficient Software Prebuilds\n\nPyTorch's [pip and conda builds](https://pytorch.org/get-started/locally/#start-locally) come prebuilt with the cuda toolkit \nwhich is enough to run PyTorch, but it is insufficient if you need to build cuda extensions.\n\nAt times, additional efforts may be required to pre-build some components. For instance, if you're using libraries like `apex` that \ndon't come pre-compiled. In other situations figuring out how to install the right cuda toolkit system-wide can be complicated. \nTo address these scenarios PyTorch and NVIDIA released a new version of NGC docker container which already comes with \neverything prebuilt. You just need to install your programs on it, and it will run out of the box.\n\nThis approach is also useful if you want to tweak the pytorch source and/or make a new customized build.\nTo find the docker image version you want start [with PyTorch release notes](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/), \nchoose one of the latest monthly releases. Go into the release's notes for the desired release, check that the environment's \ncomponents are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go \nto the corresponding NGC page. If for some reason you get lost, here is [the index of all PyTorch NGC images](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch).\n\nNext follow the instructions to download and deploy the docker image.\n\n## Mixture of Experts\n\nSome recent papers reported a 4-5x training speedup and a faster inference by integrating\nMixture of Experts (MoE) into the Transformer models.\n\nSince it has been discovered that more parameters lead to better performance, this technique allows to increase the \nnumber of parameters by an order of magnitude without increasing training costs.\n\nIn this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function \nthat trains each expert in a balanced way depending on the input token's position in a sequence.\n\n!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2050127c-296e-4bd6-8594-50cc2cef38fc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2050127c-296e-4bd6-8594-50cc2cef38fc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3ba2c6e0-c19a-4c3a-9098-453fd20451e1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"708fa0d836cfa698919509d6eca0915033d63a9ec393c839f584d067cb263011\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7835e437-a55d-4091-8884-070bc205f005\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fad7fc434cf82aed64403587dcfba7a85f85afd4baaf54269dc232e446980511\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29745, \"end_char_idx\": 32427, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "At times, additional efforts may be required to pre-build some components. For instance, if you're using libraries like `apex` that \ndon't come pre-compiled. In other situations figuring out how to install the right cuda toolkit system-wide can be complicated. \nTo address these scenarios PyTorch and NVIDIA released a new version of NGC docker container which already comes with \neverything prebuilt. You just need to install your programs on it, and it will run out of the box.\n\nThis approach is also useful if you want to tweak the pytorch source and/or make a new customized build.\nTo find the docker image version you want start [with PyTorch release notes](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/), \nchoose one of the latest monthly releases. Go into the release's notes for the desired release, check that the environment's \ncomponents are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go \nto the corresponding NGC page. If for some reason you get lost, here is [the index of all PyTorch NGC images](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch).\n\nNext follow the instructions to download and deploy the docker image.\n\n## Mixture of Experts\n\nSome recent papers reported a 4-5x training speedup and a faster inference by integrating\nMixture of Experts (MoE) into the Transformer models.\n\nSince it has been discovered that more parameters lead to better performance, this technique allows to increase the \nnumber of parameters by an order of magnitude without increasing training costs.\n\nIn this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function \nthat trains each expert in a balanced way depending on the input token's position in a sequence.\n\n![MoE Transformer 2x block](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perf-moe-transformer.png)\n\n(source: [GLAM](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html))\n\nYou can find exhaustive details and comparison tables in the papers listed at the end of this section.\n\nThe main drawback of this approach is that it requires staggering amounts of GPU memory - almost an order of magnitude \nlarger than its dense equivalent. Various distillation and approaches are proposed to how to overcome the much higher memory requirements.\n\nThere is direct trade-off though, you can use just a few experts with a 2-3x smaller base model instead of dozens or \nhundreds experts leading to a 5x smaller model and thus increase the training speed moderately while increasing the \nmemory requirements moderately as well."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7835e437-a55d-4091-8884-070bc205f005
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7835e437-a55d-4091-8884-070bc205f005\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2050127c-296e-4bd6-8594-50cc2cef38fc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"65f9d35fca1713302e6bd6ddb84dba498398f6b6189b88891f1b507275c884bb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fbd9d825-c7b5-422c-b347-f43042c6f25b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d1685d545f4b828a3ebabcad8413e0c9f89e6f169ba9d67a2e692d9127d8ef81\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30525, \"end_char_idx\": 33971, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "Go into the release's notes for the desired release, check that the environment's \ncomponents are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go \nto the corresponding NGC page. If for some reason you get lost, here is [the index of all PyTorch NGC images](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch).\n\nNext follow the instructions to download and deploy the docker image.\n\n## Mixture of Experts\n\nSome recent papers reported a 4-5x training speedup and a faster inference by integrating\nMixture of Experts (MoE) into the Transformer models.\n\nSince it has been discovered that more parameters lead to better performance, this technique allows to increase the \nnumber of parameters by an order of magnitude without increasing training costs.\n\nIn this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function \nthat trains each expert in a balanced way depending on the input token's position in a sequence.\n\n![MoE Transformer 2x block](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perf-moe-transformer.png)\n\n(source: [GLAM](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html))\n\nYou can find exhaustive details and comparison tables in the papers listed at the end of this section.\n\nThe main drawback of this approach is that it requires staggering amounts of GPU memory - almost an order of magnitude \nlarger than its dense equivalent. Various distillation and approaches are proposed to how to overcome the much higher memory requirements.\n\nThere is direct trade-off though, you can use just a few experts with a 2-3x smaller base model instead of dozens or \nhundreds experts leading to a 5x smaller model and thus increase the training speed moderately while increasing the \nmemory requirements moderately as well.\n\nMost related papers and implementations are built around Tensorflow/TPUs:\n\n- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668)\n- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961)\n- [GLaM: Generalist Language Model (GLaM)](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html)\n\nAnd for Pytorch DeepSpeed has built one as well: [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://arxiv.org/abs/2201.05596), [Mixture of Experts](https://www.deepspeed.ai/tutorials/mixture-of-experts/) - blog posts:  [1](https://www.microsoft.com/en-us/research/blog/deepspeed-powers-8x-larger-moe-model-training-with-high-performance/), [2](https://www.microsoft.com/en-us/research/publication/scalable-and-efficient-moe-training-for-multitask-multilingual-models/) and specific deployment with large transformer-based natural language generation models: [blog post](https://www.deepspeed.ai/2021/12/09/deepspeed-moe-nlg.html), [Megatron-Deepspeed branch](https://github.com/microsoft/Megatron-DeepSpeed/tree/moe-training).\n\n## Using PyTorch native attention and Flash Attention\n\nPyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fbd9d825-c7b5-422c-b347-f43042c6f25b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fbd9d825-c7b5-422c-b347-f43042c6f25b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32235eb0-3add-4442-a981-82a445254d5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7835e437-a55d-4091-8884-070bc205f005\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_one\", \"title\": \"Methods and tools for efficient training on a single GPU\", \"tokens\": 7716, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e23b78d849f4c72b46dde6c31648b5a4d0dcba81f6cd4a57a715c238a255f785\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32429, \"end_char_idx\": 34460, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32235eb0-3add-4442-a981-82a445254d5b", "doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "ref_doc_id": "32235eb0-3add-4442-a981-82a445254d5b", "chroma:document": "Most related papers and implementations are built around Tensorflow/TPUs:\n\n- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668)\n- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961)\n- [GLaM: Generalist Language Model (GLaM)](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html)\n\nAnd for Pytorch DeepSpeed has built one as well: [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://arxiv.org/abs/2201.05596), [Mixture of Experts](https://www.deepspeed.ai/tutorials/mixture-of-experts/) - blog posts:  [1](https://www.microsoft.com/en-us/research/blog/deepspeed-powers-8x-larger-moe-model-training-with-high-performance/), [2](https://www.microsoft.com/en-us/research/publication/scalable-and-efficient-moe-training-for-multitask-multilingual-models/) and specific deployment with large transformer-based natural language generation models: [blog post](https://www.deepspeed.ai/2021/12/09/deepspeed-moe-nlg.html), [Megatron-Deepspeed branch](https://github.com/microsoft/Megatron-DeepSpeed/tree/moe-training).\n\n## Using PyTorch native attention and Flash Attention\n\nPyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood. SDPA support is currently being added natively in Transformers and is used by default for `torch>=2.1.1` when an implementation is available. Please refer to [PyTorch scaled dot product attention](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention) for a list of supported models and more details.\n\nCheck out this [blogpost](https://pytorch.org/blog/out-of-the-box-acceleration/) to learn more about acceleration and memory-savings with SDPA."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f2e7d165-487a-424e-9478-35d8d44fbe31
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f2e7d165-487a-424e-9478-35d8d44fbe31\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e30ed985-91a2-42e3-84b0-d1de8c9f2a3d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eaf39928c104f616403f50a4ec03f81bd2d1ff8b54de892267a1798b2f08099e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3287, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "ref_doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "chroma:document": "<!---\nCopyright 2022 The HuggingFace Team. All rights reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n\u26a0\ufe0f Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be\nrendered properly in your Markdown viewer.\n\n-->\n\n# Installation\n\nInstall \ud83e\udd17 Transformers for whichever deep learning library you're working with, setup your cache, and optionally configure \ud83e\udd17 Transformers to run offline.\n\n\ud83e\udd17 Transformers is tested on Python 3.6+, PyTorch 1.1.0+, TensorFlow 2.0+, and Flax. Follow the installation instructions below for the deep learning library you are using:\n\n* [PyTorch](https://pytorch.org/get-started/locally/) installation instructions.\n* [TensorFlow 2.0](https://www.tensorflow.org/install/pip) installation instructions.\n* [Flax](https://flax.readthedocs.io/en/latest/) installation instructions.\n\n## Install with pip\n\nYou should install \ud83e\udd17 Transformers in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're unfamiliar with Python virtual environments, take a look at this [guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). A virtual environment makes it easier to manage different projects, and avoid compatibility issues between dependencies.\n\nStart by creating a virtual environment in your project directory:\n\n```bash\npython -m venv .env\n```\n\nActivate the virtual environment. On Linux and MacOs:\n\n```bash\nsource .env/bin/activate\n```\nActivate Virtual environment on Windows\n\n```bash\n.env/Scripts/activate\n```\n\nNow you're ready to install \ud83e\udd17 Transformers with the following command:\n\n```bash\npip install transformers\n```\n\nFor CPU-support only, you can conveniently install \ud83e\udd17 Transformers and a deep learning library in one line. For example, install \ud83e\udd17 Transformers and PyTorch with:\n\n```bash\npip install 'transformers[torch]'\n```\n\n\ud83e\udd17 Transformers and TensorFlow 2.0:\n\n```bash\npip install 'transformers[tf-cpu]'\n```\n\n<Tip warning={true}>\n\nM1 / ARM Users\n\nYou will need to install the following before installing TensorFLow 2.0\n```bash\nbrew install cmake\nbrew install pkg-config\n```\n\n</Tip>\n\n\ud83e\udd17 Transformers and Flax:\n\n```bash\npip install 'transformers[flax]'\n```\n\nFinally, check if \ud83e\udd17 Transformers has been properly installed by running the following command. It will download a pretrained model:\n\n```bash\npython -c \"from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))\"\n```\n\nThen print out the label and score:\n\n```bash\n[{'label': 'POSITIVE', 'score': 0.9998704791069031}]\n```\n\n## Install from source\n\nInstall \ud83e\udd17 Transformers from source with the following command:\n\n```bash\npip install git+https://github.com/huggingface/transformers\n```\n\nThis command installs the bleeding edge `main` version rather than the latest `stable` version."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e30ed985-91a2-42e3-84b0-d1de8c9f2a3d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e30ed985-91a2-42e3-84b0-d1de8c9f2a3d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f2e7d165-487a-424e-9478-35d8d44fbe31\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"edbeed3a858acf3816698e4bcbd4bbc950fcef1a4e31cc378113d8a2b0516739\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"72a54998-78b9-41f6-8073-647ce1c72a9e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b5d0d21984db14aa1802940e6d744bd6ef0f340de8005559300114e9e2b4ccb3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1642, \"end_char_idx\": 4848, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "ref_doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "chroma:document": "A virtual environment makes it easier to manage different projects, and avoid compatibility issues between dependencies.\n\nStart by creating a virtual environment in your project directory:\n\n```bash\npython -m venv .env\n```\n\nActivate the virtual environment. On Linux and MacOs:\n\n```bash\nsource .env/bin/activate\n```\nActivate Virtual environment on Windows\n\n```bash\n.env/Scripts/activate\n```\n\nNow you're ready to install \ud83e\udd17 Transformers with the following command:\n\n```bash\npip install transformers\n```\n\nFor CPU-support only, you can conveniently install \ud83e\udd17 Transformers and a deep learning library in one line. For example, install \ud83e\udd17 Transformers and PyTorch with:\n\n```bash\npip install 'transformers[torch]'\n```\n\n\ud83e\udd17 Transformers and TensorFlow 2.0:\n\n```bash\npip install 'transformers[tf-cpu]'\n```\n\n<Tip warning={true}>\n\nM1 / ARM Users\n\nYou will need to install the following before installing TensorFLow 2.0\n```bash\nbrew install cmake\nbrew install pkg-config\n```\n\n</Tip>\n\n\ud83e\udd17 Transformers and Flax:\n\n```bash\npip install 'transformers[flax]'\n```\n\nFinally, check if \ud83e\udd17 Transformers has been properly installed by running the following command. It will download a pretrained model:\n\n```bash\npython -c \"from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))\"\n```\n\nThen print out the label and score:\n\n```bash\n[{'label': 'POSITIVE', 'score': 0.9998704791069031}]\n```\n\n## Install from source\n\nInstall \ud83e\udd17 Transformers from source with the following command:\n\n```bash\npip install git+https://github.com/huggingface/transformers\n```\n\nThis command installs the bleeding edge `main` version rather than the latest `stable` version. The `main` version is useful for staying up-to-date with the latest developments. For instance, if a bug has been fixed since the last official release but a new release hasn't been rolled out yet. However, this means the `main` version may not always be stable. We strive to keep the `main` version operational, and most issues are usually resolved within a few hours or a day. If you run into a problem, please open an [Issue](https://github.com/huggingface/transformers/issues) so we can fix it even sooner!\n\nCheck if \ud83e\udd17 Transformers has been properly installed by running the following command:\n\n```bash\npython -c \"from transformers import pipeline; print(pipeline('sentiment-analysis')('I love you'))\"\n```\n\n## Editable install\n\nYou will need an editable install if you'd like to:\n\n* Use the `main` version of the source code.\n* Contribute to \ud83e\udd17 Transformers and need to test changes in the code.\n\nClone the repository and install \ud83e\udd17 Transformers with the following commands:\n\n```bash\ngit clone https://github.com/huggingface/transformers.git\ncd transformers\npip install -e .\n```\n\nThese commands will link the folder you cloned the repository to and your Python library paths. Python will now look inside the folder you cloned to in addition to the normal library paths. For example, if your Python packages are typically installed in `~/anaconda3/envs/main/lib/python3.7/site-packages/`, Python will also search the folder you cloned to: `~/transformers/`.\n\n<Tip warning={true}>\n\nYou must keep the `transformers` folder if you want to keep using the library."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 72a54998-78b9-41f6-8073-647ce1c72a9e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"72a54998-78b9-41f6-8073-647ce1c72a9e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e30ed985-91a2-42e3-84b0-d1de8c9f2a3d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"49cd58cce254bbf930fc90689541a1fdc14bafdb9586051469ab4a2e7bf81bd5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c0d2791b-ba1a-4c78-8178-a77ff315bb87\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cbb13bdfa3d6f1604e84f17fb180aa9742a49094c04025c6eae711f93f8d4164\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3288, \"end_char_idx\": 6442, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "ref_doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "chroma:document": "The `main` version is useful for staying up-to-date with the latest developments. For instance, if a bug has been fixed since the last official release but a new release hasn't been rolled out yet. However, this means the `main` version may not always be stable. We strive to keep the `main` version operational, and most issues are usually resolved within a few hours or a day. If you run into a problem, please open an [Issue](https://github.com/huggingface/transformers/issues) so we can fix it even sooner!\n\nCheck if \ud83e\udd17 Transformers has been properly installed by running the following command:\n\n```bash\npython -c \"from transformers import pipeline; print(pipeline('sentiment-analysis')('I love you'))\"\n```\n\n## Editable install\n\nYou will need an editable install if you'd like to:\n\n* Use the `main` version of the source code.\n* Contribute to \ud83e\udd17 Transformers and need to test changes in the code.\n\nClone the repository and install \ud83e\udd17 Transformers with the following commands:\n\n```bash\ngit clone https://github.com/huggingface/transformers.git\ncd transformers\npip install -e .\n```\n\nThese commands will link the folder you cloned the repository to and your Python library paths. Python will now look inside the folder you cloned to in addition to the normal library paths. For example, if your Python packages are typically installed in `~/anaconda3/envs/main/lib/python3.7/site-packages/`, Python will also search the folder you cloned to: `~/transformers/`.\n\n<Tip warning={true}>\n\nYou must keep the `transformers` folder if you want to keep using the library.\n\n</Tip>\n\nNow you can easily update your clone to the latest version of \ud83e\udd17 Transformers with the following command:\n\n```bash\ncd ~/transformers/\ngit pull\n```\n\nYour Python environment will find the `main` version of \ud83e\udd17 Transformers on the next run.\n\n## Install with conda\n\nInstall from the conda channel `conda-forge`:\n\n```bash\nconda install conda-forge::transformers\n```\n\n## Cache setup\n\nPretrained models are downloaded and locally cached at: `~/.cache/huggingface/hub`. This is the default directory given by the shell environment variable `TRANSFORMERS_CACHE`. On Windows, the default directory is given by `C:\\Users\\username\\.cache\\huggingface\\hub`. You can change the shell environment variables shown below - in order of priority - to specify a different cache directory:\n\n1. Shell environment variable (default): `HUGGINGFACE_HUB_CACHE` or `TRANSFORMERS_CACHE`.\n2. Shell environment variable: `HF_HOME`.\n3. Shell environment variable: `XDG_CACHE_HOME` + `/huggingface`.\n\n<Tip>\n\n\ud83e\udd17 Transformers will use the shell environment variables `PYTORCH_TRANSFORMERS_CACHE` or `PYTORCH_PRETRAINED_BERT_CACHE` if you are coming from an earlier iteration of this library and have set those environment variables, unless you specify the shell environment variable `TRANSFORMERS_CACHE`.\n\n</Tip>\n\n## Offline mode\n\nRun \ud83e\udd17 Transformers in a firewalled or offline environment with locally cached files by setting the environment variable `HF_HUB_OFFLINE=1`.\n\n<Tip>\n\nAdd [\ud83e\udd17 Datasets](https://huggingface.co/docs/datasets/) to your offline training workflow with the environment variable `HF_DATASETS_OFFLINE=1`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c0d2791b-ba1a-4c78-8178-a77ff315bb87
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c0d2791b-ba1a-4c78-8178-a77ff315bb87\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"72a54998-78b9-41f6-8073-647ce1c72a9e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e86ca2a92d70d424ce72416abdcee943a039b0f26dc74b4f8593a7c184fe53b9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a8274ade-cf07-446f-8121-2d31bea3d0d4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3a564744caca7f9bdad5df12c335ba36f51fa3013d4c26b05c2167990d7e2dc5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4850, \"end_char_idx\": 7996, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "ref_doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "chroma:document": "</Tip>\n\nNow you can easily update your clone to the latest version of \ud83e\udd17 Transformers with the following command:\n\n```bash\ncd ~/transformers/\ngit pull\n```\n\nYour Python environment will find the `main` version of \ud83e\udd17 Transformers on the next run.\n\n## Install with conda\n\nInstall from the conda channel `conda-forge`:\n\n```bash\nconda install conda-forge::transformers\n```\n\n## Cache setup\n\nPretrained models are downloaded and locally cached at: `~/.cache/huggingface/hub`. This is the default directory given by the shell environment variable `TRANSFORMERS_CACHE`. On Windows, the default directory is given by `C:\\Users\\username\\.cache\\huggingface\\hub`. You can change the shell environment variables shown below - in order of priority - to specify a different cache directory:\n\n1. Shell environment variable (default): `HUGGINGFACE_HUB_CACHE` or `TRANSFORMERS_CACHE`.\n2. Shell environment variable: `HF_HOME`.\n3. Shell environment variable: `XDG_CACHE_HOME` + `/huggingface`.\n\n<Tip>\n\n\ud83e\udd17 Transformers will use the shell environment variables `PYTORCH_TRANSFORMERS_CACHE` or `PYTORCH_PRETRAINED_BERT_CACHE` if you are coming from an earlier iteration of this library and have set those environment variables, unless you specify the shell environment variable `TRANSFORMERS_CACHE`.\n\n</Tip>\n\n## Offline mode\n\nRun \ud83e\udd17 Transformers in a firewalled or offline environment with locally cached files by setting the environment variable `HF_HUB_OFFLINE=1`.\n\n<Tip>\n\nAdd [\ud83e\udd17 Datasets](https://huggingface.co/docs/datasets/) to your offline training workflow with the environment variable `HF_DATASETS_OFFLINE=1`.\n\n</Tip>\n\n```bash\nHF_DATASETS_OFFLINE=1 HF_HUB_OFFLINE=1 \\\npython examples/pytorch/translation/run_translation.py --model_name_or_path google-t5/t5-small --dataset_name wmt16 --dataset_config ro-en ...\n```\n\nThis script should run without hanging or waiting to timeout because it won't attempt to download the model from the Hub.\n\nYou can also bypass loading a model from the Hub from each [`~PreTrainedModel.from_pretrained`] call with the [`local_files_only`] parameter. When set to `True`, only local files are loaded:\n\n```py\nfrom transformers import T5Model\n\nmodel = T5Model.from_pretrained(\"./path/to/local/directory\", local_files_only=True)\n```\n\n### Fetch models and tokenizers to use offline\n\nAnother option for using \ud83e\udd17 Transformers offline is to download the files ahead of time, and then point to their local path when you need to use them offline. There are three ways to do this:\n\n* Download a file through the user interface on the [Model Hub](https://huggingface.co/models) by clicking on the \u2193 icon.\n\n    ![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)\n\n* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:\n\n    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:\n\n    ```py\n    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n\n    >>> tokenizer = AutoTokenizer.from_pretrained(\"bigscience/T0_3B\")\n    >>> model = AutoModelForSeq2SeqLM.from_pretrained(\"bigscience/T0_3B\")\n    ```\n\n    2."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a8274ade-cf07-446f-8121-2d31bea3d0d4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a8274ade-cf07-446f-8121-2d31bea3d0d4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c0d2791b-ba1a-4c78-8178-a77ff315bb87\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2c921bc83867404130a6f4a1d72c6144c5310c9e81ba762ae24fc13b42c79202\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ac9067c8-3e9a-44a8-91b0-71b1a30659c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6777ec7304a04bb36cc1a727105977ae48e32e9dc0a90fa456c3835e1f28a445\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6444, \"end_char_idx\": 8960, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "ref_doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "chroma:document": "</Tip>\n\n```bash\nHF_DATASETS_OFFLINE=1 HF_HUB_OFFLINE=1 \\\npython examples/pytorch/translation/run_translation.py --model_name_or_path google-t5/t5-small --dataset_name wmt16 --dataset_config ro-en ...\n```\n\nThis script should run without hanging or waiting to timeout because it won't attempt to download the model from the Hub.\n\nYou can also bypass loading a model from the Hub from each [`~PreTrainedModel.from_pretrained`] call with the [`local_files_only`] parameter. When set to `True`, only local files are loaded:\n\n```py\nfrom transformers import T5Model\n\nmodel = T5Model.from_pretrained(\"./path/to/local/directory\", local_files_only=True)\n```\n\n### Fetch models and tokenizers to use offline\n\nAnother option for using \ud83e\udd17 Transformers offline is to download the files ahead of time, and then point to their local path when you need to use them offline. There are three ways to do this:\n\n* Download a file through the user interface on the [Model Hub](https://huggingface.co/models) by clicking on the \u2193 icon.\n\n    ![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)\n\n* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:\n\n    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:\n\n    ```py\n    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n\n    >>> tokenizer = AutoTokenizer.from_pretrained(\"bigscience/T0_3B\")\n    >>> model = AutoModelForSeq2SeqLM.from_pretrained(\"bigscience/T0_3B\")\n    ```\n\n    2. Save your files to a specified directory with [`PreTrainedModel.save_pretrained`]:\n\n    ```py\n    >>> tokenizer.save_pretrained(\"./your/path/bigscience_t0\")\n    >>> model.save_pretrained(\"./your/path/bigscience_t0\")\n    ```\n\n    3. Now when you're offline, reload your files with [`PreTrainedModel.from_pretrained`] from the specified directory:\n\n    ```py\n    >>> tokenizer = AutoTokenizer.from_pretrained(\"./your/path/bigscience_t0\")\n    >>> model = AutoModel.from_pretrained(\"./your/path/bigscience_t0\")\n    ```\n\n* Programmatically download files with the [huggingface_hub](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub) library:\n\n    1. Install the `huggingface_hub` library in your virtual environment:\n\n    ```bash\n    python -m pip install huggingface_hub\n    ```\n\n    2. Use the [`hf_hub_download`](https://huggingface.co/docs/hub/adding-a-library#download-files-from-the-hub) function to download a file to a specific path."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ac9067c8-3e9a-44a8-91b0-71b1a30659c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ac9067c8-3e9a-44a8-91b0-71b1a30659c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a8274ade-cf07-446f-8121-2d31bea3d0d4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/installation\", \"title\": \"Installation\", \"tokens\": 2277, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a56826f2469bf7b74763375b6a766647a543c36607c0c7ba7b353a508e039c43\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7460, \"end_char_idx\": 9702, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "ref_doc_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "chroma:document": "![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)\n\n* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:\n\n    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:\n\n    ```py\n    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n\n    >>> tokenizer = AutoTokenizer.from_pretrained(\"bigscience/T0_3B\")\n    >>> model = AutoModelForSeq2SeqLM.from_pretrained(\"bigscience/T0_3B\")\n    ```\n\n    2. Save your files to a specified directory with [`PreTrainedModel.save_pretrained`]:\n\n    ```py\n    >>> tokenizer.save_pretrained(\"./your/path/bigscience_t0\")\n    >>> model.save_pretrained(\"./your/path/bigscience_t0\")\n    ```\n\n    3. Now when you're offline, reload your files with [`PreTrainedModel.from_pretrained`] from the specified directory:\n\n    ```py\n    >>> tokenizer = AutoTokenizer.from_pretrained(\"./your/path/bigscience_t0\")\n    >>> model = AutoModel.from_pretrained(\"./your/path/bigscience_t0\")\n    ```\n\n* Programmatically download files with the [huggingface_hub](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub) library:\n\n    1. Install the `huggingface_hub` library in your virtual environment:\n\n    ```bash\n    python -m pip install huggingface_hub\n    ```\n\n    2. Use the [`hf_hub_download`](https://huggingface.co/docs/hub/adding-a-library#download-files-from-the-hub) function to download a file to a specific path. For example, the following command downloads the `config.json` file from the [T0](https://huggingface.co/bigscience/T0_3B) model to your desired path:\n\n    ```py\n    >>> from huggingface_hub import hf_hub_download\n\n    >>> hf_hub_download(repo_id=\"bigscience/T0_3B\", filename=\"config.json\", cache_dir=\"./your/path/bigscience_t0\")\n    ```\n\nOnce your file is downloaded and locally cached, specify it's local path to load and use it:\n\n```py\n>>> from transformers import AutoConfig\n\n>>> config = AutoConfig.from_pretrained(\"./your/path/bigscience_t0/config.json\")\n```\n\n<Tip>\n\nSee the [How to download files from the Hub](https://huggingface.co/docs/hub/how-to-downstream) section for more details on downloading files stored on the Hub.\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f639b09f-0616-42fd-884f-31bfd0cecf22
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f639b09f-0616-42fd-884f-31bfd0cecf22\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d4f9fc61-9a38-4a78-be11-4b65d3521275\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c8a33ebb-980c-4e3e-9bb7-18a205d0349d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cae151ad16374d59543be6e232b84e8c332430214e522f972eef428170d42e81\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3417, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "ref_doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "chroma:document": "<!---\nCopyright 2020 The HuggingFace Team. All rights reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n\u26a0\ufe0f Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be\nrendered properly in your Markdown viewer.\n\n-->\n\n# Checks on a Pull Request\n\nWhen you open a pull request on \ud83e\udd17 Transformers, a fair number of checks will be run to make sure the patch you are adding is not breaking anything existing. Those checks are of four types:\n- regular tests\n- documentation build\n- code and documentation style\n- general repository consistency\n\nIn this document, we will take a stab at explaining what those various checks are and the reason behind them, as well as how to debug them locally if one of them fails on your PR.\n\nNote that, ideally, they require you to have a dev install:\n\n```bash\npip install transformers[dev]\n```\n\nor for an editable install:\n\n```bash\npip install -e .[dev]\n```\n\ninside the Transformers repo. Since the number of optional dependencies of Transformers has grown a lot, it's possible you don't manage to get all of them. If the dev install fails, make sure to install the Deep Learning framework you are working with (PyTorch, TensorFlow and/or Flax) then do\n\n```bash\npip install transformers[quality]\n```\n\nor for an editable install:\n\n```bash\npip install -e .[quality]\n```\n\n\n## Tests\n\nAll the jobs that begin with `ci/circleci: run_tests_` run parts of the Transformers testing suite. Each of those jobs focuses on a part of the library in a certain environment: for instance `ci/circleci: run_tests_pipelines_tf` runs the pipelines test in an environment where TensorFlow only is installed.\n\nNote that to avoid running tests when there is no real change in the modules they are testing, only part of the test suite is run each time: a utility is run to determine the differences in the library between before and after the PR (what GitHub shows you in the \"Files changes\" tab) and picks the tests impacted by that diff. That utility can be run locally with:\n\n```bash\npython utils/tests_fetcher.py\n```\n\nfrom the root of the Transformers repo. It will:\n\n1. Check for each file in the diff if the changes are in the code or only in comments or docstrings. Only the files with real code changes are kept.\n2. Build an internal map that gives for each file of the source code of the library all the files it recursively impacts. Module A is said to impact module B if module B imports module A. For the recursive impact, we need a chain of modules going from module A to module B in which each module imports the previous one.\n3. Apply this map on the files gathered in step 1, which  gives us the list of model files impacted by the PR.\n4. Map each of those files to their corresponding test file(s) and get the list of tests to run.\n\nWhen executing the script locally, you should get the results of step 1, 3 and 4 printed and thus know which tests are run."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c8a33ebb-980c-4e3e-9bb7-18a205d0349d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c8a33ebb-980c-4e3e-9bb7-18a205d0349d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d4f9fc61-9a38-4a78-be11-4b65d3521275\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f639b09f-0616-42fd-884f-31bfd0cecf22\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f138789cd51e560264017a8f806beed1d2f3f2a36f388b03ab28636e9bfb6b68\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"027f133b-b3ac-4711-b7cc-c07485be2b42\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"16a310c00d2fef1fe9a44664b28695bae9b1556ce3af39ee3954139a115f73b2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1838, \"end_char_idx\": 5082, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "ref_doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "chroma:document": "## Tests\n\nAll the jobs that begin with `ci/circleci: run_tests_` run parts of the Transformers testing suite. Each of those jobs focuses on a part of the library in a certain environment: for instance `ci/circleci: run_tests_pipelines_tf` runs the pipelines test in an environment where TensorFlow only is installed.\n\nNote that to avoid running tests when there is no real change in the modules they are testing, only part of the test suite is run each time: a utility is run to determine the differences in the library between before and after the PR (what GitHub shows you in the \"Files changes\" tab) and picks the tests impacted by that diff. That utility can be run locally with:\n\n```bash\npython utils/tests_fetcher.py\n```\n\nfrom the root of the Transformers repo. It will:\n\n1. Check for each file in the diff if the changes are in the code or only in comments or docstrings. Only the files with real code changes are kept.\n2. Build an internal map that gives for each file of the source code of the library all the files it recursively impacts. Module A is said to impact module B if module B imports module A. For the recursive impact, we need a chain of modules going from module A to module B in which each module imports the previous one.\n3. Apply this map on the files gathered in step 1, which  gives us the list of model files impacted by the PR.\n4. Map each of those files to their corresponding test file(s) and get the list of tests to run.\n\nWhen executing the script locally, you should get the results of step 1, 3 and 4 printed and thus know which tests are run. The script will also create a file named `test_list.txt` which contains the list of tests to run, and you can run them locally with the following command:\n\n```bash\npython -m pytest -n 8 --dist=loadfile -rA -s $(cat test_list.txt)\n```\n\nJust in case anything slipped through the cracks, the full test suite is also run daily.\n\n## Documentation build\n\nThe `build_pr_documentation` job builds and generates a preview of the documentation to make sure everything looks okay once your PR is merged. A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on **Details** next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the `toctree`.\n\nIf you're interested in building or previewing the documentation locally, take a look at the [`README.md`](https://github.com/huggingface/transformers/tree/main/docs) in the docs folder.\n\n## Code and documentation style\n\nCode formatting is applied to all the source files, the examples and the tests using `black` and `ruff`. We also have a custom tool taking care of the formatting of docstrings and `rst` files (`utils/style_doc.py`), as well as the order of the lazy imports performed in the Transformers `__init__.py` files (`utils/custom_init_isort.py`). All of this can be launched by executing\n\n```bash\nmake style\n```\n\nThe CI checks those have been applied inside the `ci/circleci: check_code_quality` check. It also runs `ruff`, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 027f133b-b3ac-4711-b7cc-c07485be2b42
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"027f133b-b3ac-4711-b7cc-c07485be2b42\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d4f9fc61-9a38-4a78-be11-4b65d3521275\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c8a33ebb-980c-4e3e-9bb7-18a205d0349d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7c4263641f20018567083d22e48b4d3e836cac8cf7dabf8917f46a3f2d228784\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f082a685-eab2-4a31-8518-def55af32ef4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"de20bfcf29b1ccbde76088ab3b3afc431cf2d082830fb92581db07adf31ce4ac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3418, \"end_char_idx\": 5585, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "ref_doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "chroma:document": "The script will also create a file named `test_list.txt` which contains the list of tests to run, and you can run them locally with the following command:\n\n```bash\npython -m pytest -n 8 --dist=loadfile -rA -s $(cat test_list.txt)\n```\n\nJust in case anything slipped through the cracks, the full test suite is also run daily.\n\n## Documentation build\n\nThe `build_pr_documentation` job builds and generates a preview of the documentation to make sure everything looks okay once your PR is merged. A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on **Details** next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the `toctree`.\n\nIf you're interested in building or previewing the documentation locally, take a look at the [`README.md`](https://github.com/huggingface/transformers/tree/main/docs) in the docs folder.\n\n## Code and documentation style\n\nCode formatting is applied to all the source files, the examples and the tests using `black` and `ruff`. We also have a custom tool taking care of the formatting of docstrings and `rst` files (`utils/style_doc.py`), as well as the order of the lazy imports performed in the Transformers `__init__.py` files (`utils/custom_init_isort.py`). All of this can be launched by executing\n\n```bash\nmake style\n```\n\nThe CI checks those have been applied inside the `ci/circleci: check_code_quality` check. It also runs `ruff`, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used. To run that check locally, use\n\n```bash\nmake quality\n```\n\nThis can take a lot of time, so to run the same thing on only the files you modified in the current branch, run\n\n```bash\nmake fixup\n```\n\nThis last command will also run all the additional checks for the repository consistency. Let's have a look at them.\n\n## Repository consistency\n\nThis regroups all the tests to make sure your PR leaves the repository in a good state, and is performed by the `ci/circleci: check_repository_consistency` check."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f082a685-eab2-4a31-8518-def55af32ef4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f082a685-eab2-4a31-8518-def55af32ef4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d4f9fc61-9a38-4a78-be11-4b65d3521275\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"027f133b-b3ac-4711-b7cc-c07485be2b42\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b45b903e5ed4df7e2cc133e89661d3c260dfba8545e838b382fb947309dfe679\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2c6ed006-ff3c-4c41-9f91-b17dba92b3a0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c2305b09c0845a1cd234086ef51fd1574f2924d76b81af7b56eed2572f606667\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3911, \"end_char_idx\": 7747, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "ref_doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "chroma:document": "A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on **Details** next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the `toctree`.\n\nIf you're interested in building or previewing the documentation locally, take a look at the [`README.md`](https://github.com/huggingface/transformers/tree/main/docs) in the docs folder.\n\n## Code and documentation style\n\nCode formatting is applied to all the source files, the examples and the tests using `black` and `ruff`. We also have a custom tool taking care of the formatting of docstrings and `rst` files (`utils/style_doc.py`), as well as the order of the lazy imports performed in the Transformers `__init__.py` files (`utils/custom_init_isort.py`). All of this can be launched by executing\n\n```bash\nmake style\n```\n\nThe CI checks those have been applied inside the `ci/circleci: check_code_quality` check. It also runs `ruff`, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used. To run that check locally, use\n\n```bash\nmake quality\n```\n\nThis can take a lot of time, so to run the same thing on only the files you modified in the current branch, run\n\n```bash\nmake fixup\n```\n\nThis last command will also run all the additional checks for the repository consistency. Let's have a look at them.\n\n## Repository consistency\n\nThis regroups all the tests to make sure your PR leaves the repository in a good state, and is performed by the `ci/circleci: check_repository_consistency` check. You can locally run that check by executing the following:\n\n```bash\nmake repo-consistency\n```\n\nThis checks that:\n\n- All objects added to the init are documented (performed by `utils/check_repo.py`)\n- All `__init__.py` files have the same content in their two sections (performed by `utils/check_inits.py`)\n- All code identified as a copy from another module is consistent with the original (performed by `utils/check_copies.py`)\n- All configuration classes have at least one valid checkpoint mentioned in their docstrings (performed by `utils/check_config_docstrings.py`)\n- All configuration classes only contain attributes that are used in corresponding modeling files (performed by `utils/check_config_attributes.py`)\n- The translations of the READMEs and the index of the doc have the same model list as the main README (performed by `utils/check_copies.py`)\n- The auto-generated tables in the documentation are up to date (performed by `utils/check_table.py`)\n- The library has all objects available even if not all optional dependencies are installed (performed by `utils/check_dummies.py`)\n- All docstrings properly document the arguments in the signature of the object (performed by `utils/check_docstrings.py`)\n\nShould this check fail, the first two items require manual fixing, the last four can be fixed automatically for you by running the command\n\n```bash\nmake fix-copies\n```\n\nAdditional checks concern PRs that add new models, mainly that:\n\n- All models added are in an Auto-mapping (performed by `utils/check_repo.py`)\n<!-- TODO Sylvain, add a check that makes sure the common tests are implemented.-->\n- All models are properly tested (performed by `utils/check_repo.py`)\n\n<!-- TODO Sylvain, add the following\n- All models are added to the main README, inside the main doc\n- All checkpoints used actually exist on the Hub\n\n-->\n\n### Check copies\n\nSince the Transformers library is very opinionated with respect to model code, and each model should fully be implemented in a single file without relying on other models, we have added a mechanism that checks whether a copy of the code of a layer of a given model stays consistent with the original."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2c6ed006-ff3c-4c41-9f91-b17dba92b3a0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2c6ed006-ff3c-4c41-9f91-b17dba92b3a0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d4f9fc61-9a38-4a78-be11-4b65d3521275\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f082a685-eab2-4a31-8518-def55af32ef4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0a046faad9bc1a3e3e960d35da49e86ada96a632fcd555e1da78feb6f6563544\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f8deb596-a676-4792-982a-c4c41cd916e4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f94a0cf9b286bf13b56756048eb29e027b83b10654d866954da7a4f982981675\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7748, \"end_char_idx\": 10116, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "ref_doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "chroma:document": "This way, when there is a bug fix, we can see all other impacted models and choose to trickle down the modification or break the copy.\n\n<Tip>\n\nIf a file is a full copy of another file, you should register it in the constant `FULL_COPIES` of `utils/check_copies.py`.\n\n</Tip>\n\nThis mechanism relies on comments of the form `# Copied from xxx`. The `xxx` should contain the whole path to the class of function which is being copied below. For instance, `RobertaSelfOutput` is a direct copy of the `BertSelfOutput` class, so you can see [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L289) it has a comment:\n\n```py\n# Copied from transformers.models.bert.modeling_bert.BertSelfOutput\n```\n\nNote that instead of applying this to a whole class, you can apply it to the relevant methods that are copied from. For instance [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L598) you can see how `RobertaPreTrainedModel._init_weights` is copied from the same method in `BertPreTrainedModel` with the comment:\n\n```py\n# Copied from transformers.models.bert.modeling_bert.BertPreTrainedModel._init_weights\n```\n\nSometimes the copy is exactly the same except for names: for instance in `RobertaAttention`, we use `RobertaSelfAttention` insted of `BertSelfAttention` but other than that, the code is exactly the same. This is why `# Copied from` supports simple string replacements with the following syntax: `Copied from xxx with foo->bar`. This means the code is copied with all instances of `foo` being replaced by `bar`. You can see how it used [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L304C1-L304C86) in `RobertaAttention` with the comment:\n\n```py\n# Copied from transformers.models.bert.modeling_bert.BertAttention with Bert->Roberta\n```\n\nNote that there shouldn't be any spaces around the arrow (unless that space is part of the pattern to replace of course).\n\nYou can add several patterns separated by a comma. For instance here `CamemberForMaskedLM` is a direct copy of `RobertaForMaskedLM` with two replacements: `Roberta` to `Camembert` and `ROBERTA` to `CAMEMBERT`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f8deb596-a676-4792-982a-c4c41cd916e4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f8deb596-a676-4792-982a-c4c41cd916e4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d4f9fc61-9a38-4a78-be11-4b65d3521275\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2c6ed006-ff3c-4c41-9f91-b17dba92b3a0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pr_checks\", \"title\": \"Checks on a Pull Request\", \"tokens\": 2764, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f9cde03352f5688a4271baa6cf177ee13193507e88d141ccd356cb4ee4c20ed6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9250, \"end_char_idx\": 11612, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "ref_doc_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "chroma:document": "This is why `# Copied from` supports simple string replacements with the following syntax: `Copied from xxx with foo->bar`. This means the code is copied with all instances of `foo` being replaced by `bar`. You can see how it used [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L304C1-L304C86) in `RobertaAttention` with the comment:\n\n```py\n# Copied from transformers.models.bert.modeling_bert.BertAttention with Bert->Roberta\n```\n\nNote that there shouldn't be any spaces around the arrow (unless that space is part of the pattern to replace of course).\n\nYou can add several patterns separated by a comma. For instance here `CamemberForMaskedLM` is a direct copy of `RobertaForMaskedLM` with two replacements: `Roberta` to `Camembert` and `ROBERTA` to `CAMEMBERT`. You can see [here](https://github.com/huggingface/transformers/blob/15082a9dc6950ecae63a0d3e5060b2fc7f15050a/src/transformers/models/camembert/modeling_camembert.py#L929) this is done with the comment:\n\n```py\n# Copied from transformers.models.roberta.modeling_roberta.RobertaForMaskedLM with Roberta->Camembert, ROBERTA->CAMEMBERT\n```\n\nIf the order matters (because one of the replacements might conflict with a previous one), the replacements are executed from left to right.\n\n<Tip>\n\nIf the replacements change the formatting (if you replace a short name by a very long name for instance), the copy is checked after applying the auto-formatter.\n\n</Tip>\n\nAnother way when the patterns are just different casings of the same replacement (with an uppercased and a lowercased variants) is just to add the option `all-casing`. [Here](https://github.com/huggingface/transformers/blob/15082a9dc6950ecae63a0d3e5060b2fc7f15050a/src/transformers/models/mobilebert/modeling_mobilebert.py#L1237) is an example in `MobileBertForSequenceClassification` with the comment:\n\n```py\n# Copied from transformers.models.bert.modeling_bert.BertForSequenceClassification with Bert->MobileBert all-casing\n```\n\nIn this case, the code is copied from `BertForSequenceClassification` by replacing:\n- `Bert` by `MobileBert` (for instance when using `MobileBertModel` in the init)\n- `bert` by `mobilebert` (for instance when defining `self.mobilebert`)\n- `BERT` by `MOBILEBERT` (in the constant `MOBILEBERT_INPUTS_DOCSTRING`)"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5d9ffa25-e619-417d-8c3a-1946b8436c30
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tflite", "title": "Export to TFLite", "tokens": 763, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5d9ffa25-e619-417d-8c3a-1946b8436c30\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tflite\", \"title\": \"Export to TFLite\", \"tokens\": 763, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9129df82-e582-491b-933d-20847bfa58ec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tflite\", \"title\": \"Export to TFLite\", \"tokens\": 763, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0130f2681f69d6ce6f652cbe10af6485aa43a241bb98b616df984578248e01fd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2358, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9129df82-e582-491b-933d-20847bfa58ec", "doc_id": "9129df82-e582-491b-933d-20847bfa58ec", "ref_doc_id": "9129df82-e582-491b-933d-20847bfa58ec", "chroma:document": "# Export to TFLite\n\n[TensorFlow Lite](https://www.tensorflow.org/lite/guide) is a lightweight framework for deploying machine learning models \non resource-constrained devices, such as mobile phones, embedded systems, and Internet of Things (IoT) devices. \nTFLite is designed to optimize and run models efficiently on these devices with limited computational power, memory, and \npower consumption.\nA TensorFlow Lite model is represented in a special efficient portable format identified by the `.tflite` file extension. \n\n\ud83e\udd17 Optimum offers functionality to export \ud83e\udd17 Transformers models to TFLite through the `exporters.tflite` module. \nFor the list of supported model architectures, please refer to [\ud83e\udd17 Optimum documentation](https://huggingface.co/docs/optimum/exporters/tflite/overview).\n\nTo export a model to TFLite, install the required dependencies:\n \n```bash\npip install optimum[exporters-tf]\n```\n\nTo check out all available arguments, refer to the [\ud83e\udd17 Optimum docs](https://huggingface.co/docs/optimum/main/en/exporters/tflite/usage_guides/export_a_model), \nor view help in command line:\n\n```bash\noptimum-cli export tflite --help\n```\n\nTo export a model's checkpoint from the \ud83e\udd17 Hub, for example, `google-bert/bert-base-uncased`, run the following command:\n\n```bash\noptimum-cli export tflite --model google-bert/bert-base-uncased --sequence_length 128 bert_tflite/\n```\n\nYou should see the logs indicating progress and showing where the resulting `model.tflite` is saved, like this:\n\n```bash\nValidating TFLite model...\n\t-[\u2713] TFLite model output names match reference model (logits)\n\t- Validating TFLite Model output \"logits\":\n\t\t-[\u2713] (1, 128, 30522) matches (1, 128, 30522)\n\t\t-[x] values not close enough, max diff: 5.817413330078125e-05 (atol: 1e-05)\nThe TensorFlow Lite export succeeded with the warning: The maximum absolute difference between the output of the reference model and the TFLite exported model is not within the set tolerance 1e-05:\n- logits: max diff = 5.817413330078125e-05.\n The exported model was saved at: bert_tflite\n ```\n\nThe example above illustrates exporting a checkpoint from \ud83e\udd17 Hub. When exporting a local model, first make sure that you \nsaved both the model's weights and tokenizer files in the same directory (`local_path`). When using CLI, pass the \n`local_path` to the `model` argument instead of the checkpoint name on \ud83e\udd17 Hub."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 062977ad-9651-48c3-8a44-b7f50a27c2f3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"062977ad-9651-48c3-8a44-b7f50a27c2f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cbad4e38-e6e4-45c6-b69c-09e2e6459183\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aa61503410636d6cdc5e54d4f157ad12b345e897dc6150a6fb5978cb9be0970c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1766, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "# Optimize inference using torch.compile()\n\nThis guide aims to provide a benchmark on the inference speed-ups introduced with [`torch.compile()`](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html)\u00a0for [computer vision models in \ud83e\udd17 Transformers](https://huggingface.co/models?pipeline_tag=image-classification&library=transformers&sort=trending).\n\n## Benefits of torch.compile\n   \nDepending on the model and the GPU, `torch.compile()` yields up to 30% speed-up during inference. To use `torch.compile()`, simply install any version of `torch` above 2.0. \n\nCompiling a model takes time, so it's useful if you are compiling the model only once instead of every time you infer.\nTo compile any computer vision model of your choice, call `torch.compile()` on the model as shown below:\n\n```diff\nfrom transformers import AutoModelForImageClassification\n\nmodel = AutoModelForImageClassification.from_pretrained(MODEL_ID).to(\"cuda\")\n+ model = torch.compile(model)\n```\n\n`compile()`\u00a0comes with multiple modes for compiling, which essentially differ in compilation time and inference overhead. `max-autotune`\u00a0takes longer than `reduce-overhead`\u00a0but results in faster inference. Default mode is fastest for compilation but is not as efficient compared to `reduce-overhead` for inference time. In this guide, we used the default mode. You can learn more about it [here](https://pytorch.org/get-started/pytorch-2.0/#user-experience).\n\nWe benchmarked `torch.compile` with different computer vision models, tasks, types of hardware, and batch sizes on `torch`\u00a0version 2.0.1.\n\n## Benchmarking code \n\nBelow you can find the benchmarking code for each task. We warm up the GPU before inference and take the mean time of 300 inferences, using the same image each time."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cbad4e38-e6e4-45c6-b69c-09e2e6459183
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cbad4e38-e6e4-45c6-b69c-09e2e6459183\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"062977ad-9651-48c3-8a44-b7f50a27c2f3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8fbf805a473a6237681974baf73e8bb26fc5bc896a04eb2676d04d0fa95569fe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"83987ac8-1bf1-411c-8d5e-d45e4afe98c0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"64ec2e10906c55dd8900250c0c554b47201ea338ede7369b9d133c03c74e5237\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 367, \"end_char_idx\": 3469, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "## Benefits of torch.compile\n   \nDepending on the model and the GPU, `torch.compile()` yields up to 30% speed-up during inference. To use `torch.compile()`, simply install any version of `torch` above 2.0. \n\nCompiling a model takes time, so it's useful if you are compiling the model only once instead of every time you infer.\nTo compile any computer vision model of your choice, call `torch.compile()` on the model as shown below:\n\n```diff\nfrom transformers import AutoModelForImageClassification\n\nmodel = AutoModelForImageClassification.from_pretrained(MODEL_ID).to(\"cuda\")\n+ model = torch.compile(model)\n```\n\n`compile()`\u00a0comes with multiple modes for compiling, which essentially differ in compilation time and inference overhead. `max-autotune`\u00a0takes longer than `reduce-overhead`\u00a0but results in faster inference. Default mode is fastest for compilation but is not as efficient compared to `reduce-overhead` for inference time. In this guide, we used the default mode. You can learn more about it [here](https://pytorch.org/get-started/pytorch-2.0/#user-experience).\n\nWe benchmarked `torch.compile` with different computer vision models, tasks, types of hardware, and batch sizes on `torch`\u00a0version 2.0.1.\n\n## Benchmarking code \n\nBelow you can find the benchmarking code for each task. We warm up the GPU before inference and take the mean time of 300 inferences, using the same image each time.\n\n### Image Classification with ViT\n\n```python \nimport torch\nfrom PIL import Image\nimport requests\nimport numpy as np\nfrom transformers import AutoImageProcessor, AutoModelForImageClassification\n\nurl = 'http://images.cocodataset.org/val2017/000000039769.jpg'\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprocessor = AutoImageProcessor.from_pretrained(\"google/vit-base-patch16-224\")\nmodel = AutoModelForImageClassification.from_pretrained(\"google/vit-base-patch16-224\").to(\"cuda\")\nmodel = torch.compile(model)\n\nprocessed_input = processor(image, return_tensors='pt').to(device=\"cuda\")\n\nwith torch.no_grad():\n    _ = model(**processed_input)\n\n```\n\n#### Object Detection with DETR\n\n```python \nfrom transformers import AutoImageProcessor, AutoModelForObjectDetection\n\nprocessor = AutoImageProcessor.from_pretrained(\"facebook/detr-resnet-50\")\nmodel = AutoModelForObjectDetection.from_pretrained(\"facebook/detr-resnet-50\").to(\"cuda\")\nmodel = torch.compile(model)\n\ntexts = [\"a photo of a cat\", \"a photo of a dog\"]\ninputs = processor(text=texts, images=image, return_tensors=\"pt\").to(\"cuda\")\n\nwith torch.no_grad():\n    _ = model(**inputs)\n```\n\n#### Image Segmentation with Segformer\n\n```python \nfrom transformers import SegformerImageProcessor, SegformerForSemanticSegmentation\n\nprocessor = SegformerImageProcessor.from_pretrained(\"nvidia/segformer-b0-finetuned-ade-512-512\")\nmodel = SegformerForSemanticSegmentation.from_pretrained(\"nvidia/segformer-b0-finetuned-ade-512-512\").to(\"cuda\")\nmodel = torch.compile(model)\nseg_inputs = processor(images=image, return_tensors=\"pt\").to(\"cuda\")\n\nwith torch.no_grad():\n    _ = model(**seg_inputs)\n```\n\nBelow you can find the list of the models we benchmarked."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 83987ac8-1bf1-411c-8d5e-d45e4afe98c0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"83987ac8-1bf1-411c-8d5e-d45e4afe98c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cbad4e38-e6e4-45c6-b69c-09e2e6459183\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b839ab54ceaef4e92c2765bfa74109c09f646f7f91efc794fd279b9eee543ba2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e381c62b-0e91-4727-81a4-4c386a8adb2a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5325ed3cddc9941ad793c36ebc7e01c0bb28b7a5a8c55c7040a129ef35e1f35f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3471, \"end_char_idx\": 5618, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "**Image Classification** \n- [google/vit-base-patch16-224](https://huggingface.co/google/vit-base-patch16-224)\n- [microsoft/beit-base-patch16-224-pt22k-ft22k](https://huggingface.co/microsoft/beit-base-patch16-224-pt22k-ft22k)\n- [facebook/convnext-large-224](https://huggingface.co/facebook/convnext-large-224)\n- [microsoft/resnet-50](https://huggingface.co/microsoft/resnet-50)\n\n**Image Segmentation** \n- [nvidia/segformer-b0-finetuned-ade-512-512](https://huggingface.co/nvidia/segformer-b0-finetuned-ade-512-512)\n- [facebook/mask2former-swin-tiny-coco-panoptic](https://huggingface.co/facebook/mask2former-swin-tiny-coco-panoptic)\n- [facebook/maskformer-swin-base-ade](https://huggingface.co/facebook/maskformer-swin-base-ade)\n- [google/deeplabv3_mobilenet_v2_1.0_513](https://huggingface.co/google/deeplabv3_mobilenet_v2_1.0_513)\n\n**Object Detection** \n- [google/owlvit-base-patch32](https://huggingface.co/google/owlvit-base-patch32)\n- [facebook/detr-resnet-101](https://huggingface.co/facebook/detr-resnet-101)\n- [microsoft/conditional-detr-resnet-50](https://huggingface.co/microsoft/conditional-detr-resnet-50)\n\nBelow you can find visualization of inference durations with and without `torch.compile()`\u00a0and percentage improvements for each model in different hardware and batch sizes. \n\n<div class=\"flex\">\n  <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/a100_batch_comp.png\" />\n  </div>\n  <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/v100_batch_comp.png\" />\n  </div>\n   <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/t4_batch_comp.png\" />\n  </div>\n</div>\n\n<div class=\"flex\">\n  <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_duration.png\" />\n  </div>\n  <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_percentage.png\" />\n  </div>\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e381c62b-0e91-4727-81a4-4c386a8adb2a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e381c62b-0e91-4727-81a4-4c386a8adb2a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"83987ac8-1bf1-411c-8d5e-d45e4afe98c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7937efdd5c76938bf67d92f454928b1c84ccdd6f278818a271708247f52cba4f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"725f6cbf-d69f-44a4-90c3-2c61808a5424\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1ee584ad0f7e7e9e75ae09c5d5c4c2876552e2eef499c4dac771ead7c1bd8ed0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4765, \"end_char_idx\": 7277, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "<div class=\"flex\">\n  <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/a100_batch_comp.png\" />\n  </div>\n  <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/v100_batch_comp.png\" />\n  </div>\n   <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/t4_batch_comp.png\" />\n  </div>\n</div>\n\n<div class=\"flex\">\n  <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_duration.png\" />\n  </div>\n  <div>\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_percentage.png\" />\n  </div>\n</div>\n\n\n![Duration Comparison on V100 with Batch Size of 1](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/v100_1_duration.png)\n\n![Percentage Improvement on T4 with Batch Size of 4](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/T4_4_percentage.png)\n\nBelow you can find inference durations in milliseconds for each model with and without `compile()`. Note that OwlViT results in OOM in larger batch sizes.\n\n### A100 (batch size: 1)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 9.325 | 7.584 | \n| Image Segmentation/Segformer | 11.759 | 10.500 |\n| Object Detection/OwlViT | 24.978 | 18.420 |\n| Image Classification/BeiT | 11.282 | 8.448 | \n| Object Detection/DETR | 34.619 | 19.040 |\n| Image Classification/ConvNeXT | 10.410 | 10.208 | \n| Image Classification/ResNet | 6.531 | 4.124 |\n| Image Segmentation/Mask2former | 60.188 | 49.117 |\n| Image Segmentation/Maskformer | 75.764 | 59.487 | \n| Image Segmentation/MobileNet | 8.583 | 3.974 |\n| Object Detection/Resnet-101 | 36.276 | 18.197 |\n| Object Detection/Conditional-DETR | 31.219 | 17.993 |\n\n\n### A100 (batch size: 4)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 14.832 | 14.499 | \n| Image Segmentation/Segformer | 18.838 | 16.476 |\n| Image Classification/BeiT | 13.205 | 13.048 | \n| Object Detection/DETR | 48.657 | 32.418|\n| Image Classification/ConvNeXT | 22.940 | 21.631 | \n| Image Classification/ResNet | 6.657 | 4."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 725f6cbf-d69f-44a4-90c3-2c61808a5424
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"725f6cbf-d69f-44a4-90c3-2c61808a5424\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e381c62b-0e91-4727-81a4-4c386a8adb2a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5875c3288d14e5f3fa87b47ec992a72e3c77ed7482a46a5ee7f3cf6b9cf49c60\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ff4761b6-fb7d-4c23-96e4-c3eaddf224ee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d6b7857f59fc78b1b22c907248693b118eb20152ccf5e14f5bcd2463ebb449eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6863, \"end_char_idx\": 8908, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "### A100 (batch size: 4)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 14.832 | 14.499 | \n| Image Segmentation/Segformer | 18.838 | 16.476 |\n| Image Classification/BeiT | 13.205 | 13.048 | \n| Object Detection/DETR | 48.657 | 32.418|\n| Image Classification/ConvNeXT | 22.940 | 21.631 | \n| Image Classification/ResNet | 6.657 | 4.268 |\n| Image Segmentation/Mask2former | 74.277 | 61.781 |\n| Image Segmentation/Maskformer | 180.700 | 159.116 | \n| Image Segmentation/MobileNet | 14.174 | 8.515 |\n| Object Detection/Resnet-101 | 68.101 | 44.998 |\n| Object Detection/Conditional-DETR | 56.470 | 35.552 |\n\n### A100 (batch size: 16)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 40.944 | 40.010 | \n| Image Segmentation/Segformer | 37.005 | 31.144 |\n| Image Classification/BeiT | 41.854 | 41.048 | \n| Object Detection/DETR | 164.382 | 161.902 |\n| Image Classification/ConvNeXT | 82.258 | 75.561 | \n| Image Classification/ResNet | 7.018 | 5.024 |\n| Image Segmentation/Mask2former | 178.945 | 154.814 |\n| Image Segmentation/Maskformer | 638.570 | 579.826 | \n| Image Segmentation/MobileNet | 51.693 | 30.310 |\n| Object Detection/Resnet-101 | 232.887 | 155.021 |\n| Object Detection/Conditional-DETR | 180.491 | 124.032 |\n\n### V100 (batch size: 1)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 10.495 | 6.00 | \n| Image Segmentation/Segformer | 13.321 | 5.862 | \n| Object Detection/OwlViT | 25.769 | 22.395 | \n| Image Classification/BeiT | 11.347 | 7.234 | \n| Object Detection/DETR | 33.951 | 19.388 |\n| Image Classification/ConvNeXT | 11.623 | 10.412 | \n| Image Classification/ResNet | 6.484 | 3.820 |\n| Image Segmentation/Mask2former | 64.640 | 49.873 |\n| Image Segmentation/Maskformer | 95.532 | 72.207 | \n| Image Segmentation/MobileNet | 9.217 | 4.753 |\n| Object Detection/Resnet-101 | 52.818 | 28."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ff4761b6-fb7d-4c23-96e4-c3eaddf224ee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ff4761b6-fb7d-4c23-96e4-c3eaddf224ee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"725f6cbf-d69f-44a4-90c3-2c61808a5424\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"73fe75cd007aec1a17762800401c830c06bf08d27622b26295e5372b138b1867\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a1759262-c6e5-43de-b66d-200213355225\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"351268f5efc603dca17cf46b34f34824d971c976bcc080d2a098e9d731fe7789\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7855, \"end_char_idx\": 9930, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "382 | 161.902 |\n| Image Classification/ConvNeXT | 82.258 | 75.561 | \n| Image Classification/ResNet | 7.018 | 5.024 |\n| Image Segmentation/Mask2former | 178.945 | 154.814 |\n| Image Segmentation/Maskformer | 638.570 | 579.826 | \n| Image Segmentation/MobileNet | 51.693 | 30.310 |\n| Object Detection/Resnet-101 | 232.887 | 155.021 |\n| Object Detection/Conditional-DETR | 180.491 | 124.032 |\n\n### V100 (batch size: 1)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 10.495 | 6.00 | \n| Image Segmentation/Segformer | 13.321 | 5.862 | \n| Object Detection/OwlViT | 25.769 | 22.395 | \n| Image Classification/BeiT | 11.347 | 7.234 | \n| Object Detection/DETR | 33.951 | 19.388 |\n| Image Classification/ConvNeXT | 11.623 | 10.412 | \n| Image Classification/ResNet | 6.484 | 3.820 |\n| Image Segmentation/Mask2former | 64.640 | 49.873 |\n| Image Segmentation/Maskformer | 95.532 | 72.207 | \n| Image Segmentation/MobileNet | 9.217 | 4.753 |\n| Object Detection/Resnet-101 | 52.818 | 28.367 |\n| Object Detection/Conditional-DETR | 39.512 | 20.816 |\n\n### V100 (batch size: 4)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 15.181 | 14.501 | \n| Image Segmentation/Segformer | 16.787 | 16.188 |\n| Image Classification/BeiT | 15.171 | 14.753 | \n| Object Detection/DETR | 88.529 | 64.195 |\n| Image Classification/ConvNeXT | 29.574 | 27.085 | \n| Image Classification/ResNet | 6.109 | 4.731 |\n| Image Segmentation/Mask2former | 90.402 | 76.926 |\n| Image Segmentation/Maskformer | 234.261 | 205.456 | \n| Image Segmentation/MobileNet | 24.623 | 14.816 |\n| Object Detection/Resnet-101 | 134.672 | 101.304 |\n| Object Detection/Conditional-DETR | 97.464 | 69.739 |\n\n### V100 (batch size: 16)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 52.209 | 51.633 | \n| Image Segmentation/Segformer | 61.013 | 55.499 |\n| Image Classification/BeiT | 53.938 | 53."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a1759262-c6e5-43de-b66d-200213355225
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a1759262-c6e5-43de-b66d-200213355225\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ff4761b6-fb7d-4c23-96e4-c3eaddf224ee\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2a312f3bccb0de7a727d8e145e2da3771f124493e621b205262c7b31b7b445a4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aead2690-0d35-4d6c-b7d1-0bccc94f6a28\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4f31c039172791eb9fb95cd3aeb5850d0eaeafde8a5db06dcac79c14a7b7b692\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8899, \"end_char_idx\": 10969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "818 | 28.367 |\n| Object Detection/Conditional-DETR | 39.512 | 20.816 |\n\n### V100 (batch size: 4)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 15.181 | 14.501 | \n| Image Segmentation/Segformer | 16.787 | 16.188 |\n| Image Classification/BeiT | 15.171 | 14.753 | \n| Object Detection/DETR | 88.529 | 64.195 |\n| Image Classification/ConvNeXT | 29.574 | 27.085 | \n| Image Classification/ResNet | 6.109 | 4.731 |\n| Image Segmentation/Mask2former | 90.402 | 76.926 |\n| Image Segmentation/Maskformer | 234.261 | 205.456 | \n| Image Segmentation/MobileNet | 24.623 | 14.816 |\n| Object Detection/Resnet-101 | 134.672 | 101.304 |\n| Object Detection/Conditional-DETR | 97.464 | 69.739 |\n\n### V100 (batch size: 16)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 52.209 | 51.633 | \n| Image Segmentation/Segformer | 61.013 | 55.499 |\n| Image Classification/BeiT | 53.938 | 53.581  |\n| Object Detection/DETR | OOM | OOM |\n| Image Classification/ConvNeXT | 109.682 | 100.771 | \n| Image Classification/ResNet | 14.857 | 12.089 |\n| Image Segmentation/Mask2former | 249.605 | 222.801 |\n| Image Segmentation/Maskformer | 831.142 | 743.645 | \n| Image Segmentation/MobileNet | 93.129 | 55.365 |\n| Object Detection/Resnet-101 | 482.425 | 361.843 |\n| Object Detection/Conditional-DETR | 344.661 | 255.298 |\n\n### T4 (batch size: 1)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 16.520 | 15.786 | \n| Image Segmentation/Segformer | 16.116 | 14.205 |\n| Object Detection/OwlViT | 53.634 | 51.105 |\n| Image Classification/BeiT | 16.464 | 15.710 | \n| Object Detection/DETR | 73.100 | 53.99 |\n| Image Classification/ConvNeXT | 32.932 | 30.845 | \n| Image Classification/ResNet | 6.031 | 4.321 |\n| Image Segmentation/Mask2former | 79.192 | 66.815 |\n| Image Segmentation/Maskformer | 200.026 | 188.268 | \n| Image Segmentation/MobileNet | 18.908 | 11."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: aead2690-0d35-4d6c-b7d1-0bccc94f6a28
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"aead2690-0d35-4d6c-b7d1-0bccc94f6a28\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a1759262-c6e5-43de-b66d-200213355225\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e1031341ae5c0cf78aa6b5f508a348b63d8617c47ed85d140ea5840abb51c686\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"096f770c-16fc-4514-a0c7-646abfc36603\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a5e248a607ec8791392e67a56ac6aa4d5ecd033ff6543db0de1bdaa177e84087\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9882, \"end_char_idx\": 11994, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "499 |\n| Image Classification/BeiT | 53.938 | 53.581  |\n| Object Detection/DETR | OOM | OOM |\n| Image Classification/ConvNeXT | 109.682 | 100.771 | \n| Image Classification/ResNet | 14.857 | 12.089 |\n| Image Segmentation/Mask2former | 249.605 | 222.801 |\n| Image Segmentation/Maskformer | 831.142 | 743.645 | \n| Image Segmentation/MobileNet | 93.129 | 55.365 |\n| Object Detection/Resnet-101 | 482.425 | 361.843 |\n| Object Detection/Conditional-DETR | 344.661 | 255.298 |\n\n### T4 (batch size: 1)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 16.520 | 15.786 | \n| Image Segmentation/Segformer | 16.116 | 14.205 |\n| Object Detection/OwlViT | 53.634 | 51.105 |\n| Image Classification/BeiT | 16.464 | 15.710 | \n| Object Detection/DETR | 73.100 | 53.99 |\n| Image Classification/ConvNeXT | 32.932 | 30.845 | \n| Image Classification/ResNet | 6.031 | 4.321 |\n| Image Segmentation/Mask2former | 79.192 | 66.815 |\n| Image Segmentation/Maskformer | 200.026 | 188.268 | \n| Image Segmentation/MobileNet | 18.908 | 11.997 |\n| Object Detection/Resnet-101 | 106.622 | 82.566 |\n| Object Detection/Conditional-DETR | 77.594 | 56.984 |\n\n### T4 (batch size: 4)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 43.653 | 43.626 | \n| Image Segmentation/Segformer | 45.327 | 42.445 |\n| Image Classification/BeiT | 52.007 | 51.354 | \n| Object Detection/DETR | 277.850 | 268.003 |\n| Image Classification/ConvNeXT | 119.259 | 105.580 | \n| Image Classification/ResNet | 13.039 | 11.388 |\n| Image Segmentation/Mask2former | 201.540 | 184.670 |\n| Image Segmentation/Maskformer | 764.052 | 711.280 | \n| Image Segmentation/MobileNet | 74.289 | 48.677 |\n| Object Detection/Resnet-101 | 421.859 | 357.614 |\n| Object Detection/Conditional-DETR | 289.002 | 226.945 |\n\n### T4 (batch size: 16)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 163.914 | 160.907 | \n| Image Segmentation/Segformer | 192."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 096f770c-16fc-4514-a0c7-646abfc36603
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"096f770c-16fc-4514-a0c7-646abfc36603\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aead2690-0d35-4d6c-b7d1-0bccc94f6a28\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"065acf7504420687129dacbf17fae248ab7375bd4abe9b64b5eb796bf7b0a241\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e53e29ac-b016-4336-9826-647c1c2ed03c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a9f9b97c53781c583bd40baa27483b6c8dd5a0cbb4cbb2e48db9ee7bacd832a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10960, \"end_char_idx\": 12699, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "908 | 11.997 |\n| Object Detection/Resnet-101 | 106.622 | 82.566 |\n| Object Detection/Conditional-DETR | 77.594 | 56.984 |\n\n### T4 (batch size: 4)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 43.653 | 43.626 | \n| Image Segmentation/Segformer | 45.327 | 42.445 |\n| Image Classification/BeiT | 52.007 | 51.354 | \n| Object Detection/DETR | 277.850 | 268.003 |\n| Image Classification/ConvNeXT | 119.259 | 105.580 | \n| Image Classification/ResNet | 13.039 | 11.388 |\n| Image Segmentation/Mask2former | 201.540 | 184.670 |\n| Image Segmentation/Maskformer | 764.052 | 711.280 | \n| Image Segmentation/MobileNet | 74.289 | 48.677 |\n| Object Detection/Resnet-101 | 421.859 | 357.614 |\n| Object Detection/Conditional-DETR | 289.002 | 226.945 |\n\n### T4 (batch size: 16)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 163.914 | 160.907 | \n| Image Segmentation/Segformer | 192.412 | 163.620 |\n| Image Classification/BeiT | 188.978 | 187.976 | \n| Object Detection/DETR | OOM | OOM |\n| Image Classification/ConvNeXT | 422.886 | 388.078 | \n| Image Classification/ResNet | 44.114 | 37.604 |\n| Image Segmentation/Mask2former | 756.337 | 695.291 |\n| Image Segmentation/Maskformer | 2842.940 | 2656.88 | \n| Image Segmentation/MobileNet | 299.003 | 201.942 |\n| Object Detection/Resnet-101 |  1619.505 | 1262.758 | \n| Object Detection/Conditional-DETR | 1137.513 | 897.390|\n\n## PyTorch Nightly\nWe also benchmarked on PyTorch nightly (2.1.0dev, find the wheel [here](https://download.pytorch.org/whl/nightly/cu118)) and observed improvement in latency both for uncompiled and compiled models."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e53e29ac-b016-4336-9826-647c1c2ed03c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e53e29ac-b016-4336-9826-647c1c2ed03c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"096f770c-16fc-4514-a0c7-646abfc36603\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7da60978d7bf020d299934d26df73878bfb5f7cd8c9d3006d01ccdc9e3a45d8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b701e71-4452-4722-b523-856897587ff6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9192f96ee7c0c80ad4248351255391c240126ba3a4eda7f3d7e4a8853ce4b890\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11557, \"end_char_idx\": 14060, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "670 |\n| Image Segmentation/Maskformer | 764.052 | 711.280 | \n| Image Segmentation/MobileNet | 74.289 | 48.677 |\n| Object Detection/Resnet-101 | 421.859 | 357.614 |\n| Object Detection/Conditional-DETR | 289.002 | 226.945 |\n\n### T4 (batch size: 16)\n\n| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|\n| Image Classification/ViT | 163.914 | 160.907 | \n| Image Segmentation/Segformer | 192.412 | 163.620 |\n| Image Classification/BeiT | 188.978 | 187.976 | \n| Object Detection/DETR | OOM | OOM |\n| Image Classification/ConvNeXT | 422.886 | 388.078 | \n| Image Classification/ResNet | 44.114 | 37.604 |\n| Image Segmentation/Mask2former | 756.337 | 695.291 |\n| Image Segmentation/Maskformer | 2842.940 | 2656.88 | \n| Image Segmentation/MobileNet | 299.003 | 201.942 |\n| Object Detection/Resnet-101 |  1619.505 | 1262.758 | \n| Object Detection/Conditional-DETR | 1137.513 | 897.390|\n\n## PyTorch Nightly\nWe also benchmarked on PyTorch nightly (2.1.0dev, find the wheel [here](https://download.pytorch.org/whl/nightly/cu118)) and observed improvement in latency both for uncompiled and compiled models. \n\n### A100\n\n| **Task/Model** | **Batch Size** | **torch 2.0 - no compile** | **torch 2.0 -<br> compile** |\n|:---:|:---:|:---:|:---:|\n| Image Classification/BeiT | Unbatched | 12.462 | 6.954 | \n| Image Classification/BeiT | 4 | 14.109 | 12.851 | \n| Image Classification/BeiT | 16 | 42.179 | 42.147 | \n| Object Detection/DETR | Unbatched | 30.484 | 15.221 |\n| Object Detection/DETR | 4 | 46.816 | 30.942 |\n| Object Detection/DETR | 16 | 163.749 | 163.706  |\n\n### T4\n\n| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|:---:|\n| Image Classification/BeiT | Unbatched | 14.408 | 14.052 | \n| Image Classification/BeiT | 4 | 47.381 | 46.604 | \n| Image Classification/BeiT | 16 | 42.179 | 42.147  | \n| Object Detection/DETR | Unbatched | 68.382 | 53.481 |\n| Object Detection/DETR | 4 | 269.615 | 204.785 |\n| Object Detection/DETR | 16 | OOM | OOM   |\n\n### V100\n\n| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|:---:|\n| Image Classification/BeiT | Unbatched | 13.477 | 7.926 | \n| Image Classification/BeiT | 4 | 15.103 | 14.378 | \n| Image Classification/BeiT | 16 | 52.517 | 51.691  | \n| Object Detection/DETR | Unbatched | 28.706 | 19.077 |\n| Object Detection/DETR | 4 | 88.402 | 62.949|\n| Object Detection/DETR | 16 | OOM | OOM  |"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4b701e71-4452-4722-b523-856897587ff6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4b701e71-4452-4722-b523-856897587ff6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f7c5d2e-a02d-4508-8758-9c1afedcbda3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e53e29ac-b016-4336-9826-647c1c2ed03c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_torch_compile\", \"title\": \"Optimize inference using torch.compile()\", \"tokens\": 5184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1d7fc368e0f3e07325b89f9a52a2bec94c5cafcfe536aab584fcf4f149f8ded0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14063, \"end_char_idx\": 15405, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "ref_doc_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "chroma:document": "## Reduce Overhead\nWe benchmarked `reduce-overhead` compilation mode for A100 and T4 in Nightly.\n\n### A100\n\n| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |\n|:---:|:---:|:---:|:---:|\n| Image Classification/ConvNeXT | Unbatched | 11.758 | 7.335 | \n| Image Classification/ConvNeXT | 4 | 23.171 | 21.490 | \n| Image Classification/ResNet | Unbatched | 7.435 | 3.801 | \n| Image Classification/ResNet | 4 | 7.261 | 2.187 | \n| Object Detection/Conditional-DETR | Unbatched | 32.823 | 11.627  | \n| Object Detection/Conditional-DETR | 4 | 50.622 | 33.831  | \n| Image Segmentation/MobileNet | Unbatched | 9.869 | 4.244 |\n| Image Segmentation/MobileNet | 4 | 14.385 | 7.946 |\n\n\n### T4\n\n| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** | \n|:---:|:---:|:---:|:---:|\n| Image Classification/ConvNeXT | Unbatched | 32.137 | 31.84 | \n| Image Classification/ConvNeXT | 4 | 120.944 | 110.209 | \n| Image Classification/ResNet | Unbatched | 9.761 | 7.698 | \n| Image Classification/ResNet | 4 | 15.215 | 13.871 | \n| Object Detection/Conditional-DETR | Unbatched | 72.150 | 57.660  | \n| Object Detection/Conditional-DETR | 4 | 301.494 | 247.543  | \n| Image Segmentation/MobileNet | Unbatched | 22.266 | 19.339  |\n| Image Segmentation/MobileNet | 4 | 78.311 | 50.983 |"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e1c10abb-0f51-4120-9152-de4da0f73ca3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_special", "title": "PyTorch training on Apple silicon", "tokens": 771, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e1c10abb-0f51-4120-9152-de4da0f73ca3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_special\", \"title\": \"PyTorch training on Apple silicon\", \"tokens\": 771, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8155c6af-0dc7-424e-b5ef-cc8211a86a5e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_special\", \"title\": \"PyTorch training on Apple silicon\", \"tokens\": 771, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3b92385ad59a6c38344c47086105ae06547252dae5793516cdd7089ed1d393e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2592, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8155c6af-0dc7-424e-b5ef-cc8211a86a5e", "doc_id": "8155c6af-0dc7-424e-b5ef-cc8211a86a5e", "ref_doc_id": "8155c6af-0dc7-424e-b5ef-cc8211a86a5e", "chroma:document": "# PyTorch training on Apple silicon\n\nPreviously, training models on a Mac was limited to the CPU only. With the release of PyTorch v1.12, you can take advantage of training models with Apple's silicon GPUs for significantly faster performance and training. This is powered in PyTorch by integrating Apple's Metal Performance Shaders (MPS) as a backend. The [MPS backend](https://pytorch.org/docs/stable/notes/mps.html) implements PyTorch operations as custom Metal shaders and places these modules on a `mps` device.\n\n<Tip warning={true}>\n\nSome PyTorch operations are not implemented in MPS yet and will throw an error. To avoid this, you should set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU kernels instead (you'll still see a `UserWarning`).\n\n<br>\n\nIf you run into any other errors, please open an issue in the [PyTorch](https://github.com/pytorch/pytorch/issues) repository because the [`Trainer`] only integrates the MPS backend.\n\n</Tip>\n\nWith the `mps` device set, you can:\n\n* train larger networks or batch sizes locally\n* reduce data retrieval latency because the GPU's unified memory architecture allows direct access to the full memory store\n* reduce costs because you don't need to train on cloud-based GPUs or add additional local GPUs\n\nGet started by making sure you have PyTorch installed. MPS acceleration is supported on macOS 12.3+.\n\n```bash\npip install torch torchvision torchaudio\n```\n\n[`TrainingArguments`] uses the `mps` device by default if it's available which means you don't need to explicitly set the device. For example, you can run the [run_glue.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py) script with the MPS backend automatically enabled without making any changes.\n\n```diff\nexport TASK_NAME=mrpc\n\npython examples/pytorch/text-classification/run_glue.py \\\n  --model_name_or_path google-bert/bert-base-cased \\\n  --task_name $TASK_NAME \\\n- --use_mps_device \\\n  --do_train \\\n  --do_eval \\\n  --max_seq_length 128 \\\n  --per_device_train_batch_size 32 \\\n  --learning_rate 2e-5 \\\n  --num_train_epochs 3 \\\n  --output_dir /tmp/$TASK_NAME/ \\\n  --overwrite_output_dir\n```\n\nBackends for [distributed setups](https://pytorch.org/docs/stable/distributed.html#backends) like `gloo` and `nccl` are not supported by the `mps` device which means you can only train on a single GPU with the MPS backend.\n\nYou can learn more about the MPS backend in the [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/) blog post."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dc9ad0de-58da-4f66-9eac-90d33d1bab28
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dc9ad0de-58da-4f66-9eac-90d33d1bab28\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f08b4827-75ed-4105-b063-d8456c28e127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78272d89-897d-494e-be2f-db594a7fc9db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e27405c2aa0699f084d144afa325ddd58ce5fd51e8c95ec25e52f539539dd320\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2778, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f08b4827-75ed-4105-b063-d8456c28e127", "doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "ref_doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "chroma:document": "# Load adapters with \ud83e\udd17 PEFT\n\n[[open-in-colab]]\n\n[Parameter-Efficient Fine Tuning (PEFT)](https://huggingface.co/blog/peft) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it. The adapters are trained to learn task-specific information. This approach has been shown to be very memory-efficient with lower compute usage while producing results comparable to a fully fine-tuned model. \n\nAdapters trained with PEFT are also usually an order of magnitude smaller than the full model, making it convenient to share, store, and load them.\n\n<div class=\"flex flex-col justify-center\">\n  <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/PEFT-hub-screenshot.png\"/>\n  <figcaption class=\"text-center\">The adapter weights for a OPTForCausalLM model stored on the Hub are only ~6MB compared to the full size of the model weights, which can be ~700MB.</figcaption>\n</div>\n\nIf you're interested in learning more about the \ud83e\udd17 PEFT library, check out the [documentation](https://huggingface.co/docs/peft/index).\n\n## Setup\n\nGet started by installing \ud83e\udd17 PEFT:\n\n```bash\npip install peft\n```\n\nIf you want to try out the brand new features, you might be interested in installing the library from source:\n\n```bash\npip install git+https://github.com/huggingface/peft.git\n```\n\n## Supported PEFT models\n\n\ud83e\udd17 Transformers natively supports some PEFT methods, meaning you can load adapter weights stored locally or on the Hub and easily run or train them with a few lines of code. The following methods are supported:\n\n- [Low Rank Adapters](https://huggingface.co/docs/peft/conceptual_guides/lora)\n- [IA3](https://huggingface.co/docs/peft/conceptual_guides/ia3)\n- [AdaLoRA](https://arxiv.org/abs/2303.10512)\n\nIf you want to use other PEFT methods, such as prompt learning or prompt tuning, or about the \ud83e\udd17 PEFT library in general, please refer to the [documentation](https://huggingface.co/docs/peft/index).\n\n\n## Load a PEFT adapter\n\nTo load and use a PEFT adapter model from \ud83e\udd17 Transformers, make sure the Hub repository or local directory contains an `adapter_config.json` file and the adapter weights, as shown in the example image above. Then you can load the PEFT adapter model using the `AutoModelFor` class. For example, to load a PEFT adapter model for causal language modeling:\n\n1. specify the PEFT model id\n2. pass it to the [`AutoModelForCausalLM`] class\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\npeft_model_id = \"ybelkada/opt-350m-lora\"\nmodel = AutoModelForCausalLM.from_pretrained(peft_model_id)\n```\n\n<Tip>\n\nYou can load a PEFT adapter with either an `AutoModelFor` class or the base model class like `OPTForCausalLM` or `LlamaForCausalLM`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 78272d89-897d-494e-be2f-db594a7fc9db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"78272d89-897d-494e-be2f-db594a7fc9db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f08b4827-75ed-4105-b063-d8456c28e127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dc9ad0de-58da-4f66-9eac-90d33d1bab28\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6d0d9305367ab16700b70ee912952630caad4fd79ce75febe6526fbd63bcb5ab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"38b2e12b-720d-4a27-8e97-be3b10da96f3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4439f1bbbb111b77b87e2ed770a99c44b135919652977be5dc179b7c81383eb8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2009, \"end_char_idx\": 4089, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f08b4827-75ed-4105-b063-d8456c28e127", "doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "ref_doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "chroma:document": "## Load a PEFT adapter\n\nTo load and use a PEFT adapter model from \ud83e\udd17 Transformers, make sure the Hub repository or local directory contains an `adapter_config.json` file and the adapter weights, as shown in the example image above. Then you can load the PEFT adapter model using the `AutoModelFor` class. For example, to load a PEFT adapter model for causal language modeling:\n\n1. specify the PEFT model id\n2. pass it to the [`AutoModelForCausalLM`] class\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\npeft_model_id = \"ybelkada/opt-350m-lora\"\nmodel = AutoModelForCausalLM.from_pretrained(peft_model_id)\n```\n\n<Tip>\n\nYou can load a PEFT adapter with either an `AutoModelFor` class or the base model class like `OPTForCausalLM` or `LlamaForCausalLM`.\n\n</Tip>\n\nYou can also load a PEFT adapter by calling the `load_adapter` method:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"facebook/opt-350m\"\npeft_model_id = \"ybelkada/opt-350m-lora\"\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\nmodel.load_adapter(peft_model_id)\n```\n\nCheck out the [API documentation](#transformers.integrations.PeftAdapterMixin) section below for more details.\n\n## Load in 8bit or 4bit\n\nThe `bitsandbytes` integration supports 8bit and 4bit precision data types, which are useful for loading large models because it saves memory (see the `bitsandbytes` integration [guide](./quantization#bitsandbytes-integration) to learn more). Add the `load_in_8bit` or `load_in_4bit` parameters to [`~PreTrainedModel.from_pretrained`] and set `device_map=\"auto\"` to effectively distribute the model to your hardware:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\npeft_model_id = \"ybelkada/opt-350m-lora\"\nmodel = AutoModelForCausalLM.from_pretrained(peft_model_id, quantization_config=BitsAndBytesConfig(load_in_8bit=True))\n```\n\n## Add a new adapter\n\nYou can use [`~peft.PeftModel.add_adapter`] to add a new adapter to a model with an existing adapter as long as the new adapter is the same type as the current one."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 38b2e12b-720d-4a27-8e97-be3b10da96f3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"38b2e12b-720d-4a27-8e97-be3b10da96f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f08b4827-75ed-4105-b063-d8456c28e127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78272d89-897d-494e-be2f-db594a7fc9db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"921b7e54569fdd0057ffeb4f21bd1d970f070bd33c62bb24ac04f5c18d85d582\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b9df4a47-9d3f-4b44-b7d2-9be9a0f16df1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e02d31294110cd2094935b3338e93851a0aa7fbf820db5e4b4ba6744047aae34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2780, \"end_char_idx\": 5167, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f08b4827-75ed-4105-b063-d8456c28e127", "doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "ref_doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "chroma:document": "</Tip>\n\nYou can also load a PEFT adapter by calling the `load_adapter` method:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"facebook/opt-350m\"\npeft_model_id = \"ybelkada/opt-350m-lora\"\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\nmodel.load_adapter(peft_model_id)\n```\n\nCheck out the [API documentation](#transformers.integrations.PeftAdapterMixin) section below for more details.\n\n## Load in 8bit or 4bit\n\nThe `bitsandbytes` integration supports 8bit and 4bit precision data types, which are useful for loading large models because it saves memory (see the `bitsandbytes` integration [guide](./quantization#bitsandbytes-integration) to learn more). Add the `load_in_8bit` or `load_in_4bit` parameters to [`~PreTrainedModel.from_pretrained`] and set `device_map=\"auto\"` to effectively distribute the model to your hardware:\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\npeft_model_id = \"ybelkada/opt-350m-lora\"\nmodel = AutoModelForCausalLM.from_pretrained(peft_model_id, quantization_config=BitsAndBytesConfig(load_in_8bit=True))\n```\n\n## Add a new adapter\n\nYou can use [`~peft.PeftModel.add_adapter`] to add a new adapter to a model with an existing adapter as long as the new adapter is the same type as the current one. For example, if you have an existing LoRA adapter attached to a model:\n\n```py\nfrom transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer\nfrom peft import LoraConfig\n\nmodel_id = \"facebook/opt-350m\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\nlora_config = LoraConfig(\n    target_modules=[\"q_proj\", \"k_proj\"],\n    init_lora_weights=False\n)\n\nmodel.add_adapter(lora_config, adapter_name=\"adapter_1\")\n```\n\nTo add a new adapter:\n\n```py\n# attach new adapter with same config\nmodel.add_adapter(lora_config, adapter_name=\"adapter_2\")\n```\n\nNow you can use [`~peft.PeftModel.set_adapter`] to set which adapter to use:\n\n```py\n# use adapter_1\nmodel.set_adapter(\"adapter_1\")\noutput = model.generate(**inputs)\nprint(tokenizer.decode(output_disabled[0], skip_special_tokens=True))\n\n# use adapter_2\nmodel.set_adapter(\"adapter_2\")\noutput_enabled = model.generate(**inputs)\nprint(tokenizer.decode(output_enabled[0], skip_special_tokens=True))\n```\n\n## Enable and disable adapters\n\nOnce you've added an adapter to a model, you can enable or disable the adapter module."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b9df4a47-9d3f-4b44-b7d2-9be9a0f16df1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b9df4a47-9d3f-4b44-b7d2-9be9a0f16df1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f08b4827-75ed-4105-b063-d8456c28e127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"38b2e12b-720d-4a27-8e97-be3b10da96f3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"38e6e2f9e0c6e85aa43a5e6082994beadd0b09fd9b900bb2d7439075ac100da8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be10172c-53c7-42d5-aec0-23c83d1c8bf5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8bc89707a9d994367f824d0c23884bd9c696dc335ef2a4491d941c57e222aafa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4090, \"end_char_idx\": 7112, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f08b4827-75ed-4105-b063-d8456c28e127", "doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "ref_doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "chroma:document": "For example, if you have an existing LoRA adapter attached to a model:\n\n```py\nfrom transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer\nfrom peft import LoraConfig\n\nmodel_id = \"facebook/opt-350m\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\nlora_config = LoraConfig(\n    target_modules=[\"q_proj\", \"k_proj\"],\n    init_lora_weights=False\n)\n\nmodel.add_adapter(lora_config, adapter_name=\"adapter_1\")\n```\n\nTo add a new adapter:\n\n```py\n# attach new adapter with same config\nmodel.add_adapter(lora_config, adapter_name=\"adapter_2\")\n```\n\nNow you can use [`~peft.PeftModel.set_adapter`] to set which adapter to use:\n\n```py\n# use adapter_1\nmodel.set_adapter(\"adapter_1\")\noutput = model.generate(**inputs)\nprint(tokenizer.decode(output_disabled[0], skip_special_tokens=True))\n\n# use adapter_2\nmodel.set_adapter(\"adapter_2\")\noutput_enabled = model.generate(**inputs)\nprint(tokenizer.decode(output_enabled[0], skip_special_tokens=True))\n```\n\n## Enable and disable adapters\n\nOnce you've added an adapter to a model, you can enable or disable the adapter module. To enable the adapter module:\n\n```py\nfrom transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer\nfrom peft import PeftConfig\n\nmodel_id = \"facebook/opt-350m\"\nadapter_model_id = \"ybelkada/opt-350m-lora\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\ntext = \"Hello\"\ninputs = tokenizer(text, return_tensors=\"pt\")\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\npeft_config = PeftConfig.from_pretrained(adapter_model_id)\n\n# to initiate with random weights\npeft_config.init_lora_weights = False\n\nmodel.add_adapter(peft_config)\nmodel.enable_adapters()\noutput = model.generate(**inputs)\n```\n\nTo disable the adapter module:\n\n```py\nmodel.disable_adapters()\noutput = model.generate(**inputs)\n```\n\n## Train a PEFT adapter\n\nPEFT adapters are supported by the [`Trainer`] class so that you can train an adapter for your specific use case. It only requires adding a few more lines of code. For example, to train a LoRA adapter:\n\n<Tip>\n\nIf you aren't familiar with fine-tuning a model with [`Trainer`], take a look at the [Fine-tune a pretrained model](training) tutorial.\n\n</Tip>\n\n1. Define your adapter configuration with the task type and hyperparameters (see [`~peft.LoraConfig`] for more details about what the hyperparameters do).\n\n```py\nfrom peft import LoraConfig\n\npeft_config = LoraConfig(\n    lora_alpha=16,\n    lora_dropout=0.1,\n    r=64,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n)\n```\n\n2. Add adapter to the model.\n\n```py\nmodel.add_adapter(peft_config)\n```\n\n3. Now you can pass the model to [`Trainer`]!\n\n```py\ntrainer = Trainer(model=model, ...)\ntrainer.train()\n```\n\nTo save your trained adapter and load it back:\n\n```py\nmodel.save_pretrained(save_dir)\nmodel = AutoModelForCausalLM.from_pretrained(save_dir)\n```\n\n## Add additional trainable layers to a PEFT adapter\n\nYou can also fine-tune additional trainable adapters on top of a model that has adapters attached by passing `modules_to_save` in your PEFT config."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: be10172c-53c7-42d5-aec0-23c83d1c8bf5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"be10172c-53c7-42d5-aec0-23c83d1c8bf5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f08b4827-75ed-4105-b063-d8456c28e127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b9df4a47-9d3f-4b44-b7d2-9be9a0f16df1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/peft\", \"title\": \"Load adapters with \\ud83e\\udd17 PEFT\", \"tokens\": 2169, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0888577565f0dff656e787a803fb0695de17fbfd15d1f8cf7210a5f89b7c21ad\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6023, \"end_char_idx\": 7882, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f08b4827-75ed-4105-b063-d8456c28e127", "doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "ref_doc_id": "f08b4827-75ed-4105-b063-d8456c28e127", "chroma:document": "It only requires adding a few more lines of code. For example, to train a LoRA adapter:\n\n<Tip>\n\nIf you aren't familiar with fine-tuning a model with [`Trainer`], take a look at the [Fine-tune a pretrained model](training) tutorial.\n\n</Tip>\n\n1. Define your adapter configuration with the task type and hyperparameters (see [`~peft.LoraConfig`] for more details about what the hyperparameters do).\n\n```py\nfrom peft import LoraConfig\n\npeft_config = LoraConfig(\n    lora_alpha=16,\n    lora_dropout=0.1,\n    r=64,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n)\n```\n\n2. Add adapter to the model.\n\n```py\nmodel.add_adapter(peft_config)\n```\n\n3. Now you can pass the model to [`Trainer`]!\n\n```py\ntrainer = Trainer(model=model, ...)\ntrainer.train()\n```\n\nTo save your trained adapter and load it back:\n\n```py\nmodel.save_pretrained(save_dir)\nmodel = AutoModelForCausalLM.from_pretrained(save_dir)\n```\n\n## Add additional trainable layers to a PEFT adapter\n\nYou can also fine-tune additional trainable adapters on top of a model that has adapters attached by passing `modules_to_save` in your PEFT config. For example, if you want to also fine-tune the lm_head on top of a model with a LoRA adapter:\n\n```py\nfrom transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer\nfrom peft import LoraConfig\n\nmodel_id = \"facebook/opt-350m\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\nlora_config = LoraConfig(\n    target_modules=[\"q_proj\", \"k_proj\"],\n    modules_to_save=[\"lm_head\"],\n)\n\nmodel.add_adapter(lora_config)\n```\n\n## API docs\n\n[[autodoc]] integrations.PeftAdapterMixin\n    - load_adapter\n    - add_adapter\n    - set_adapter\n    - disable_adapters\n    - enable_adapters\n    - active_adapters\n    - get_adapter_state_dict\n\n\n\n\n<!--\nTODO: (@younesbelkada @stevhliu)\n-   Link to PEFT docs for further details\n-   Trainer  \n-   8-bit / 4-bit examples ?\n-->"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b6da5fa2-0ebb-462d-b747-aaf58a3bbfcc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b6da5fa2-0ebb-462d-b747-aaf58a3bbfcc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c667a00-4d05-4905-948b-7c06e021fd96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ef853e4-b754-4cc8-93c8-545d9dbc8f17\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"36c01562d2d54924e66a1d73f7069c5b734782af77a28d1c786cef6dc8296979\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3041, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "ref_doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "chroma:document": "# Translation\n\n[[open-in-colab]]\n\n<Youtube id=\"1JvfrvZgi6c\"/>\n\nTranslation converts a sequence of text from one language to another. It is one of several tasks you can formulate as a sequence-to-sequence problem, a powerful framework for returning some output from an input, like translation or summarization. Translation systems are commonly used for translation between different language texts, but it can also be used for speech or some combination in between like text-to-speech or speech-to-text.\n\nThis guide will show you how to:\n\n1. Finetune [T5](https://huggingface.co/google-t5/t5-small) on the English-French subset of the [OPUS Books](https://huggingface.co/datasets/opus_books) dataset to translate English text to French.\n2. Use your finetuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/translation).\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install transformers datasets evaluate sacrebleu\n```\n\nWe encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load OPUS Books dataset\n\nStart by loading the English-French subset of the [OPUS Books](https://huggingface.co/datasets/opus_books) dataset from the \ud83e\udd17 Datasets library:\n\n```py\n>>> from datasets import load_dataset\n\n>>> books = load_dataset(\"opus_books\", \"en-fr\")\n```\n\nSplit the dataset into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n```py\n>>> books = books[\"train\"].train_test_split(test_size=0.2)\n```\n\nThen take a look at an example:\n\n```py\n>>> books[\"train\"][0]\n{'id': '90560',\n 'translation': {'en': 'But this lofty plateau measured only a few fathoms, and soon we reentered Our Element.',\n  'fr': 'Mais ce plateau \u00e9lev\u00e9 ne mesurait que quelques toises, et bient\u00f4t nous f\u00fbmes rentr\u00e9s dans notre \u00e9l\u00e9ment.'}}\n```\n\n`translation`: an English and French translation of the text.\n\n## Preprocess\n\n<Youtube id=\"XAR8jnZZuUs\"/>\n\nThe next step is to load a T5 tokenizer to process the English-French language pairs:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> checkpoint = \"google-t5/t5-small\"\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n```\n\nThe preprocessing function you want to create needs to:\n\n1. Prefix the input with a prompt so T5 knows this is a translation task. Some models capable of multiple NLP tasks require prompting for specific tasks.\n2. Set the target language (French) in the `text_target` parameter to ensure the tokenizer processes the target text correctly. If you don't set `text_target`, the tokenizer processes the target text as English.\n3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.\n\n```py\n>>> source_lang = \"en\"\n>>> target_lang = \"fr\"\n>>> prefix = \"translate English to French: \""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1ef853e4-b754-4cc8-93c8-545d9dbc8f17
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1ef853e4-b754-4cc8-93c8-545d9dbc8f17\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c667a00-4d05-4905-948b-7c06e021fd96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b6da5fa2-0ebb-462d-b747-aaf58a3bbfcc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5873fc647971bee97da66f601c8e76e51688d79aa2364b9d241d5951ff4efabb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1e29be7c-b38a-4e26-af48-e380385bf6c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f424f901e7a4e1f207adbbec5fcc63901bddeae43953d7efa8b2d8c73cd3a6aa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3044, \"end_char_idx\": 6300, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "ref_doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "chroma:document": ">>> def preprocess_function(examples):\n...     inputs = [prefix + example[source_lang] for example in examples[\"translation\"]]\n...     targets = [example[target_lang] for example in examples[\"translation\"]]\n...     model_inputs = tokenizer(inputs, text_target=targets, max_length=128, truncation=True)\n...     return model_inputs\n```\n\nTo apply the preprocessing function over the entire dataset, use \ud83e\udd17 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:\n\n```py\n>>> tokenized_books = books.map(preprocess_function, batched=True)\n```\n\nNow create a batch of examples using [`DataCollatorForSeq2Seq`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.\n\n<frameworkcontent>\n<pt>\n```py\n>>> from transformers import DataCollatorForSeq2Seq\n\n>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint)\n```\n</pt>\n<tf>\n\n```py\n>>> from transformers import DataCollatorForSeq2Seq\n\n>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint, return_tensors=\"tf\")\n```\n</tf>\n</frameworkcontent>\n\n## Evaluate\n\nIncluding a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the \ud83e\udd17 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [SacreBLEU](https://huggingface.co/spaces/evaluate-metric/sacrebleu) metric (see the \ud83e\udd17 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):\n\n```py\n>>> import evaluate\n\n>>> metric = evaluate.load(\"sacrebleu\")\n```\n\nThen create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the SacreBLEU score:\n\n```py\n>>> import numpy as np\n\n\n>>> def postprocess_text(preds, labels):\n...     preds = [pred.strip() for pred in preds]\n...     labels = [[label.strip()] for label in labels]\n\n...     return preds, labels\n\n\n>>> def compute_metrics(eval_preds):\n...     preds, labels = eval_preds\n...     if isinstance(preds, tuple):\n...         preds = preds[0]\n...     decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)\n\n...     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)\n...     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)\n\n...     decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)\n\n...     result = metric.compute(predictions=decoded_preds, references=decoded_labels)\n...     result = {\"bleu\": result[\"score\"]}\n\n...     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]\n...     result[\"gen_len\"] = np.mean(prediction_lens)\n...     result = {k: round(v, 4) for k, v in result.items()}\n...     return result\n```\n\nYour `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.\n\n## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1e29be7c-b38a-4e26-af48-e380385bf6c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1e29be7c-b38a-4e26-af48-e380385bf6c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c667a00-4d05-4905-948b-7c06e021fd96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1ef853e4-b754-4cc8-93c8-545d9dbc8f17\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"dc2d016e45c89c4ffcd54eda80e4525647259ee4f3c32a0b88a9e29c8627c1de\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d3f1319e-ead0-4e02-b006-22f56371f1ae\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b908af4f159c3e551b2b0af5e0c38554a20b4d7b9cf208c0c62e066f5fd9ce94\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4994, \"end_char_idx\": 7197, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "ref_doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "chroma:document": ">>> def postprocess_text(preds, labels):\n...     preds = [pred.strip() for pred in preds]\n...     labels = [[label.strip()] for label in labels]\n\n...     return preds, labels\n\n\n>>> def compute_metrics(eval_preds):\n...     preds, labels = eval_preds\n...     if isinstance(preds, tuple):\n...         preds = preds[0]\n...     decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)\n\n...     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)\n...     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)\n\n...     decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)\n\n...     result = metric.compute(predictions=decoded_preds, references=decoded_labels)\n...     result = {\"bleu\": result[\"score\"]}\n\n...     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]\n...     result[\"gen_len\"] = np.mean(prediction_lens)\n...     result = {k: round(v, 4) for k, v in result.items()}\n...     return result\n```\n\nYour `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.\n\n## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load T5 with [`AutoModelForSeq2SeqLM`]:\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`Seq2SeqTrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the SacreBLEU metric and save the training checkpoint.\n2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d3f1319e-ead0-4e02-b006-22f56371f1ae
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d3f1319e-ead0-4e02-b006-22f56371f1ae\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c667a00-4d05-4905-948b-7c06e021fd96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1e29be7c-b38a-4e26-af48-e380385bf6c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"56f27bfc03004b4200188a4d7503e331a70f4cda132fea82ed5a127226b5f7b4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5cb86bd0-f7a2-43a1-988b-7a5c719a9a85\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dfa70486787d27016373dbaf027cd410cb9abbf2a3f7aaf98388324fb417132b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6109, \"end_char_idx\": 8268, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "ref_doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "chroma:document": "## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load T5 with [`AutoModelForSeq2SeqLM`]:\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`Seq2SeqTrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the SacreBLEU metric and save the training checkpoint.\n2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = Seq2SeqTrainingArguments(\n...     output_dir=\"my_awesome_opus_books_model\",\n...     eval_strategy=\"epoch\",\n...     learning_rate=2e-5,\n...     per_device_train_batch_size=16,\n...     per_device_eval_batch_size=16,\n...     weight_decay=0.01,\n...     save_total_limit=3,\n...     num_train_epochs=2,\n...     predict_with_generate=True,\n...     fp16=True,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Seq2SeqTrainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=tokenized_books[\"train\"],\n...     eval_dataset=tokenized_books[\"test\"],\n...     tokenizer=tokenizer,\n...     data_collator=data_collator,\n...     compute_metrics=compute_metrics,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n<tf>\n<Tip>\n\nIf you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5cb86bd0-f7a2-43a1-988b-7a5c719a9a85
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5cb86bd0-f7a2-43a1-988b-7a5c719a9a85\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c667a00-4d05-4905-948b-7c06e021fd96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d3f1319e-ead0-4e02-b006-22f56371f1ae\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3d6e2ea9f3444122d12cd7f6d9d08fe87898c44b507c6f4afcd65408ed6fecc9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"40f566ef-9409-49f8-b39d-4d13607c907e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"60de0cecc43834dc4ca4d83458ffac3949d79a8ae69d8a6ff74334dc3e2572b7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6765, \"end_char_idx\": 9771, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "ref_doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "chroma:document": "You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the SacreBLEU metric and save the training checkpoint.\n2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = Seq2SeqTrainingArguments(\n...     output_dir=\"my_awesome_opus_books_model\",\n...     eval_strategy=\"epoch\",\n...     learning_rate=2e-5,\n...     per_device_train_batch_size=16,\n...     per_device_eval_batch_size=16,\n...     weight_decay=0.01,\n...     save_total_limit=3,\n...     num_train_epochs=2,\n...     predict_with_generate=True,\n...     fp16=True,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Seq2SeqTrainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=tokenized_books[\"train\"],\n...     eval_dataset=tokenized_books[\"test\"],\n...     tokenizer=tokenizer,\n...     data_collator=data_collator,\n...     compute_metrics=compute_metrics,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n<tf>\n<Tip>\n\nIf you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!\n\n</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import AdamWeightDecay\n\n>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)\n```\n\nThen you can load T5 with [`TFAutoModelForSeq2SeqLM`]:\n\n```py\n>>> from transformers import TFAutoModelForSeq2SeqLM\n\n>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     tokenized_books[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_test_set = model.prepare_tf_dataset(\n...     tokenized_books[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\n>>> import tensorflow as tf\n\n>>> model.compile(optimizer=optimizer)  # No loss argument!\n```\n\nThe last two things to setup before you start training is to compute the SacreBLEU metric from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 40f566ef-9409-49f8-b39d-4d13607c907e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"40f566ef-9409-49f8-b39d-4d13607c907e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c667a00-4d05-4905-948b-7c06e021fd96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5cb86bd0-f7a2-43a1-988b-7a5c719a9a85\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"99cce7458dadef385934746b057ee85f68db20486bf52b8114cf39395d2d89e5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7f56dd5f-7640-44d8-b11f-da13aafd575f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8fce709c167c86bdb02defdbfc2e7ccd937efd2908b41d93774123f4918b6eb1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8270, \"end_char_idx\": 11501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "ref_doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "chroma:document": "</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import AdamWeightDecay\n\n>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)\n```\n\nThen you can load T5 with [`TFAutoModelForSeq2SeqLM`]:\n\n```py\n>>> from transformers import TFAutoModelForSeq2SeqLM\n\n>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     tokenized_books[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_test_set = model.prepare_tf_dataset(\n...     tokenized_books[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\n>>> import tensorflow as tf\n\n>>> model.compile(optimizer=optimizer)  # No loss argument!\n```\n\nThe last two things to setup before you start training is to compute the SacreBLEU metric from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).\n\nPass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:\n\n```py\n>>> from transformers.keras_callbacks import KerasMetricCallback\n\n>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)\n```\n\nSpecify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:\n\n```py\n>>> from transformers.keras_callbacks import PushToHubCallback\n\n>>> push_to_hub_callback = PushToHubCallback(\n...     output_dir=\"my_awesome_opus_books_model\",\n...     tokenizer=tokenizer,\n... )\n```\n\nThen bundle your callbacks together:\n\n```py\n>>> callbacks = [metric_callback, push_to_hub_callback]\n```\n\nFinally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)\n```\n\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for translation, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with some text you'd like to translate to another language. For T5, you need to prefix your input depending on the task you're working on."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7f56dd5f-7640-44d8-b11f-da13aafd575f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7f56dd5f-7640-44d8-b11f-da13aafd575f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c667a00-4d05-4905-948b-7c06e021fd96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"40f566ef-9409-49f8-b39d-4d13607c907e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"20e01f7e2014e7568f27127b13221c9e05cea815045abe49d2cf9df579786114\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"de36cced-b418-4c00-9a4b-b80a37138c9e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cadd133cb6e157b37e75e40f88d9a0f8eba34b1dbf7342ad37992d5fb7483578\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10486, \"end_char_idx\": 13498, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "ref_doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "chroma:document": "Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)\n```\n\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for translation, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with some text you'd like to translate to another language. For T5, you need to prefix your input depending on the task you're working on. For translation from English to French, you should prefix your input as shown below:\n\n```py\n>>> text = \"translate English to French: Legumes share resources with nitrogen-fixing bacteria.\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for translation with your model, and pass your text to it:\n\n```py\n>>> from transformers import pipeline\n\n# Change `xx` to the language of the input and `yy` to the language of the desired output.\n# Examples: \"en\" for English, \"fr\" for French, \"de\" for German, \"es\" for Spanish, \"zh\" for Chinese, etc; translation_en_to_fr translates English to French\n# You can view all the lists of languages here - https://huggingface.co/languages\n>>> translator = pipeline(\"translation_xx_to_yy\", model=\"my_awesome_opus_books_model\")\n>>> translator(text)\n[{'translation_text': 'Legumes partagent des ressources avec des bact\u00e9ries azotantes.'}]\n```\n\nYou can also manually replicate the results of the `pipeline` if you'd like:\n\n<frameworkcontent>\n<pt>\nTokenize the text and return the `input_ids` as PyTorch tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"my_awesome_opus_books_model\")\n>>> inputs = tokenizer(text, return_tensors=\"pt\").input_ids\n```\n\nUse the [`~generation.GenerationMixin.generate`] method to create the translation. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"my_awesome_opus_books_model\")\n>>> outputs = model.generate(inputs, max_new_tokens=40, do_sample=True, top_k=30, top_p=0.95)\n```\n\nDecode the generated token ids back into text:\n\n```py\n>>> tokenizer.decode(outputs[0], skip_special_tokens=True)\n'Les lign\u00e9es partagent des ressources avec des bact\u00e9ries enfixant l'azote.'"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: de36cced-b418-4c00-9a4b-b80a37138c9e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"de36cced-b418-4c00-9a4b-b80a37138c9e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c667a00-4d05-4905-948b-7c06e021fd96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7f56dd5f-7640-44d8-b11f-da13aafd575f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/translation\", \"title\": \"Translation\", \"tokens\": 3629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a81898bc91ccc382b2212d260d0ee38bf1bbef8be1b429f7c9c6c94ed90b8ba0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12012, \"end_char_idx\": 14522, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "ref_doc_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "chroma:document": "# Examples: \"en\" for English, \"fr\" for French, \"de\" for German, \"es\" for Spanish, \"zh\" for Chinese, etc; translation_en_to_fr translates English to French\n# You can view all the lists of languages here - https://huggingface.co/languages\n>>> translator = pipeline(\"translation_xx_to_yy\", model=\"my_awesome_opus_books_model\")\n>>> translator(text)\n[{'translation_text': 'Legumes partagent des ressources avec des bact\u00e9ries azotantes.'}]\n```\n\nYou can also manually replicate the results of the `pipeline` if you'd like:\n\n<frameworkcontent>\n<pt>\nTokenize the text and return the `input_ids` as PyTorch tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"my_awesome_opus_books_model\")\n>>> inputs = tokenizer(text, return_tensors=\"pt\").input_ids\n```\n\nUse the [`~generation.GenerationMixin.generate`] method to create the translation. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"my_awesome_opus_books_model\")\n>>> outputs = model.generate(inputs, max_new_tokens=40, do_sample=True, top_k=30, top_p=0.95)\n```\n\nDecode the generated token ids back into text:\n\n```py\n>>> tokenizer.decode(outputs[0], skip_special_tokens=True)\n'Les lign\u00e9es partagent des ressources avec des bact\u00e9ries enfixant l'azote.'\n```\n</pt>\n<tf>\nTokenize the text and return the `input_ids` as TensorFlow tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"my_awesome_opus_books_model\")\n>>> inputs = tokenizer(text, return_tensors=\"tf\").input_ids\n```\n\nUse the [`~transformers.generation_tf_utils.TFGenerationMixin.generate`] method to create the translation. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.\n\n```py\n>>> from transformers import TFAutoModelForSeq2SeqLM\n\n>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(\"my_awesome_opus_books_model\")\n>>> outputs = model.generate(inputs, max_new_tokens=40, do_sample=True, top_k=30, top_p=0.95)\n```\n\nDecode the generated token ids back into text:\n\n```py\n>>> tokenizer.decode(outputs[0], skip_special_tokens=True)\n'Les lugumes partagent les ressources avec des bact\u00e9ries fixatrices d'azote.'\n```\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a2dafb25-55dc-4d9f-a356-76e3ff933e20
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a2dafb25-55dc-4d9f-a356-76e3ff933e20\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"079ba672-d6f9-484f-9dc9-608e33d4f348\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3311d53bd580533cd05c4af34f34169d9ecb33adad1a073090d19bfefc810391\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3090, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "# Document Question Answering\n\n[[open-in-colab]]\n\nDocument Question Answering, also referred to as Document Visual Question Answering, is a task that involves providing\nanswers to questions posed about document images. The input to models supporting this task is typically a combination of an image and\na question, and the output is an answer expressed in natural language. These models utilize multiple modalities, including\ntext, the positions of words (bounding boxes), and the image itself.\n\nThis guide illustrates how to:\n\n- Fine-tune [LayoutLMv2](../model_doc/layoutlmv2) on the [DocVQA dataset](https://huggingface.co/datasets/nielsr/docvqa_1200_examples_donut).\n- Use your fine-tuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-to-text)\n\n</Tip>\n\nLayoutLMv2 solves the document question-answering task by adding a question-answering head on top of the final hidden\nstates of the tokens, to predict the positions of the start and end tokens of the\nanswer. In other words, the problem is treated as extractive question answering: given the context, extract which piece\nof information answers the question. The context comes from the output of an OCR engine, here it is Google's Tesseract.\n\nBefore you begin, make sure you have all the necessary libraries installed. LayoutLMv2 depends on detectron2, torchvision and tesseract.\n\n```bash\npip install -q transformers datasets\n```\n\n```bash\npip install 'git+https://github.com/facebookresearch/detectron2.git'\npip install torchvision\n```\n\n```bash\nsudo apt install tesseract-ocr\npip install -q pytesseract\n```\n\nOnce you have installed all of the dependencies, restart your runtime.\n\nWe encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the \ud83e\udd17 Hub.\nWhen prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\nLet's define some global variables.\n\n```py\n>>> model_checkpoint = \"microsoft/layoutlmv2-base-uncased\"\n>>> batch_size = 4\n```\n\n## Load the data\n\nIn this guide we use a small sample of preprocessed DocVQA that you can find on \ud83e\udd17 Hub. If you'd like to use the full\nDocVQA dataset, you can register and download it on [DocVQA homepage](https://rrc.cvc.uab.es/?ch=17). If you do so, to\nproceed with this guide check out [how to load files into a \ud83e\udd17 dataset](https://huggingface.co/docs/datasets/loading#local-and-remote-files).\n\n```py\n>>> from datasets import load_dataset\n\n>>> dataset = load_dataset(\"nielsr/docvqa_1200_examples\")\n>>> dataset\nDatasetDict({\n    train: Dataset({\n        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],\n        num_rows: 1000\n    })\n    test: Dataset({\n        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],\n        num_rows: 200\n    })\n})\n```\n\nAs you can see, the dataset is split into train and test sets already. Take a look at a random example to familiarize\nyourself with the features."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 079ba672-d6f9-484f-9dc9-608e33d4f348
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"079ba672-d6f9-484f-9dc9-608e33d4f348\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a2dafb25-55dc-4d9f-a356-76e3ff933e20\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cac7c435be5c5d274480dd3315ba006b56fcba1820decbf902461fa6bd5b4c3f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"84176b39-8f23-431f-8c65-cf04b433c4ed\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"73b6abd76c0efa6563d6d8be28363ac08bfc38810478c4a81fcaa1c24f7e7f8b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1758, \"end_char_idx\": 4821, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the \ud83e\udd17 Hub.\nWhen prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\nLet's define some global variables.\n\n```py\n>>> model_checkpoint = \"microsoft/layoutlmv2-base-uncased\"\n>>> batch_size = 4\n```\n\n## Load the data\n\nIn this guide we use a small sample of preprocessed DocVQA that you can find on \ud83e\udd17 Hub. If you'd like to use the full\nDocVQA dataset, you can register and download it on [DocVQA homepage](https://rrc.cvc.uab.es/?ch=17). If you do so, to\nproceed with this guide check out [how to load files into a \ud83e\udd17 dataset](https://huggingface.co/docs/datasets/loading#local-and-remote-files).\n\n```py\n>>> from datasets import load_dataset\n\n>>> dataset = load_dataset(\"nielsr/docvqa_1200_examples\")\n>>> dataset\nDatasetDict({\n    train: Dataset({\n        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],\n        num_rows: 1000\n    })\n    test: Dataset({\n        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],\n        num_rows: 200\n    })\n})\n```\n\nAs you can see, the dataset is split into train and test sets already. Take a look at a random example to familiarize\nyourself with the features.\n\n```py\n>>> dataset[\"train\"].features\n```\n\nHere's what the individual fields represent:\n* `id`: the example's id\n* `image`: a PIL.Image.Image object containing the document image\n* `query`: the question string - natural language asked question, in several languages\n* `answers`: a list of correct answers provided by human annotators\n* `words` and `bounding_boxes`: the results of OCR, which we will not use here\n* `answer`: an answer matched by a different model which we will not use here\n\nLet's leave only English questions, and drop the `answer` feature which appears to contain predictions by another model.\nWe'll also take the first of the answers from the set provided by the annotators. Alternatively, you can randomly sample it.\n\n```py\n>>> updated_dataset = dataset.map(lambda example: {\"question\": example[\"query\"][\"en\"]}, remove_columns=[\"query\"])\n>>> updated_dataset = updated_dataset.map(\n...     lambda example: {\"answer\": example[\"answers\"][0]}, remove_columns=[\"answer\", \"answers\"]\n... )\n```\n\nNote that the LayoutLMv2 checkpoint that we use in this guide has been trained with `max_position_embeddings = 512` (you can\nfind this information in the [checkpoint's `config.json` file](https://huggingface.co/microsoft/layoutlmv2-base-uncased/blob/main/config.json#L18)).\nWe can truncate the examples but to avoid the situation where the answer might be at the end of a large document and end up truncated,\nhere we'll remove the few examples where the embedding is likely to end up longer than 512.\nIf most of the documents in your dataset are long, you can implement a sliding window strategy - check out [this notebook](https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb) for details."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 84176b39-8f23-431f-8c65-cf04b433c4ed
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"84176b39-8f23-431f-8c65-cf04b433c4ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"079ba672-d6f9-484f-9dc9-608e33d4f348\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cad004b9bfb0723f48bb1329c11e86bf0c9e4704ffa958fe04db04f63b4d531c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"99444eec-1809-4345-8d47-21ee72f16c07\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7c636516b9b7234d61be84065d312d4873b3300bbe82fb0cebb494b32fdf33e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3092, \"end_char_idx\": 6396, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "```py\n>>> dataset[\"train\"].features\n```\n\nHere's what the individual fields represent:\n* `id`: the example's id\n* `image`: a PIL.Image.Image object containing the document image\n* `query`: the question string - natural language asked question, in several languages\n* `answers`: a list of correct answers provided by human annotators\n* `words` and `bounding_boxes`: the results of OCR, which we will not use here\n* `answer`: an answer matched by a different model which we will not use here\n\nLet's leave only English questions, and drop the `answer` feature which appears to contain predictions by another model.\nWe'll also take the first of the answers from the set provided by the annotators. Alternatively, you can randomly sample it.\n\n```py\n>>> updated_dataset = dataset.map(lambda example: {\"question\": example[\"query\"][\"en\"]}, remove_columns=[\"query\"])\n>>> updated_dataset = updated_dataset.map(\n...     lambda example: {\"answer\": example[\"answers\"][0]}, remove_columns=[\"answer\", \"answers\"]\n... )\n```\n\nNote that the LayoutLMv2 checkpoint that we use in this guide has been trained with `max_position_embeddings = 512` (you can\nfind this information in the [checkpoint's `config.json` file](https://huggingface.co/microsoft/layoutlmv2-base-uncased/blob/main/config.json#L18)).\nWe can truncate the examples but to avoid the situation where the answer might be at the end of a large document and end up truncated,\nhere we'll remove the few examples where the embedding is likely to end up longer than 512.\nIf most of the documents in your dataset are long, you can implement a sliding window strategy - check out [this notebook](https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb) for details.\n\n```py\n>>> updated_dataset = updated_dataset.filter(lambda x: len(x[\"words\"]) + len(x[\"question\"].split()) < 512)\n```\n\nAt this point let's also remove the OCR features from this dataset. These are a result of OCR for fine-tuning a different\nmodel. They would still require some processing if we wanted to use them, as they do not match the input requirements\nof the model we use in this guide. Instead, we can use the [`LayoutLMv2Processor`] on the original data for both OCR and\ntokenization. This way we'll get the inputs that match model's expected input. If you want to process images manually,\ncheck out the [`LayoutLMv2` model documentation](../model_doc/layoutlmv2) to learn what input format the model expects.\n\n```py\n>>> updated_dataset = updated_dataset.remove_columns(\"words\")\n>>> updated_dataset = updated_dataset.remove_columns(\"bounding_boxes\")\n```\n\nFinally, the data exploration won't be complete if we don't peek at an image example.\n\n```py\n>>> updated_dataset[\"train\"][11][\"image\"]\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/docvqa_example.jpg\" alt=\"DocVQA Image Example\"/>\n </div>\n\n## Preprocess the data\n\nThe Document Question Answering task is a multimodal task, and you need to make sure that the inputs from each modality\nare preprocessed according to the model's expectations. Let's start by loading the [`LayoutLMv2Processor`], which internally combines an image processor that can handle image data and a tokenizer that can encode text data."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 99444eec-1809-4345-8d47-21ee72f16c07
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"99444eec-1809-4345-8d47-21ee72f16c07\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"84176b39-8f23-431f-8c65-cf04b433c4ed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3c5d624efd9c575748938da1e63a1b0a2432cb041797aa8f38eca2aa701d245c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"39618fb6-68bf-485b-936c-89bd83f4658f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ef191df0b3d976823f38e66de4ec3b0ed801134b0ab52fe498f73f629556a331\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4823, \"end_char_idx\": 8148, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "```py\n>>> updated_dataset = updated_dataset.filter(lambda x: len(x[\"words\"]) + len(x[\"question\"].split()) < 512)\n```\n\nAt this point let's also remove the OCR features from this dataset. These are a result of OCR for fine-tuning a different\nmodel. They would still require some processing if we wanted to use them, as they do not match the input requirements\nof the model we use in this guide. Instead, we can use the [`LayoutLMv2Processor`] on the original data for both OCR and\ntokenization. This way we'll get the inputs that match model's expected input. If you want to process images manually,\ncheck out the [`LayoutLMv2` model documentation](../model_doc/layoutlmv2) to learn what input format the model expects.\n\n```py\n>>> updated_dataset = updated_dataset.remove_columns(\"words\")\n>>> updated_dataset = updated_dataset.remove_columns(\"bounding_boxes\")\n```\n\nFinally, the data exploration won't be complete if we don't peek at an image example.\n\n```py\n>>> updated_dataset[\"train\"][11][\"image\"]\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/docvqa_example.jpg\" alt=\"DocVQA Image Example\"/>\n </div>\n\n## Preprocess the data\n\nThe Document Question Answering task is a multimodal task, and you need to make sure that the inputs from each modality\nare preprocessed according to the model's expectations. Let's start by loading the [`LayoutLMv2Processor`], which internally combines an image processor that can handle image data and a tokenizer that can encode text data.\n\n```py\n>>> from transformers import AutoProcessor\n\n>>> processor = AutoProcessor.from_pretrained(model_checkpoint)\n```\n\n### Preprocessing document images\n\nFirst, let's prepare the document images for the model with the help of the `image_processor` from the processor.\nBy default, image processor resizes the images to 224x224, makes sure they have the correct order of color channels,\napplies OCR with tesseract to get words and normalized bounding boxes. In this tutorial, all of these defaults are exactly what we need.\nWrite a function that applies the default image processing to a batch of images and returns the results of OCR.\n\n```py\n>>> image_processor = processor.image_processor\n\n\n>>> def get_ocr_words_and_boxes(examples):\n...     images = [image.convert(\"RGB\") for image in examples[\"image\"]]\n...     encoded_inputs = image_processor(images)\n\n...     examples[\"image\"] = encoded_inputs.pixel_values\n...     examples[\"words\"] = encoded_inputs.words\n...     examples[\"boxes\"] = encoded_inputs.boxes\n\n...     return examples\n```\n\nTo apply this preprocessing to the entire dataset in a fast way, use [`~datasets.Dataset.map`].\n\n```py\n>>> dataset_with_ocr = updated_dataset.map(get_ocr_words_and_boxes, batched=True, batch_size=2)\n```\n\n### Preprocessing text data\n\nOnce we have applied OCR to the images, we need to encode the text part of the dataset to prepare it for the model.\nThis involves converting the words and boxes that we got in the previous step to token-level `input_ids`, `attention_mask`,\n`token_type_ids` and `bbox`. For preprocessing text, we'll need the `tokenizer` from the processor.\n\n```py\n>>> tokenizer = processor.tokenizer\n```\n\nOn top of the preprocessing mentioned above, we also need to add the labels for the model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 39618fb6-68bf-485b-936c-89bd83f4658f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"39618fb6-68bf-485b-936c-89bd83f4658f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"99444eec-1809-4345-8d47-21ee72f16c07\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c2f66e10e73ac773e3c037d0c7640a6d28664a81bef4873ce48f3efcd3c2312e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9f316b23-bd3e-42d9-a720-b635fa3c708a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"213a74c67dd3d49267d3a4713aed4d3b93947a49030f0b9a211a41b332a28ddc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6398, \"end_char_idx\": 9193, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "```py\n>>> from transformers import AutoProcessor\n\n>>> processor = AutoProcessor.from_pretrained(model_checkpoint)\n```\n\n### Preprocessing document images\n\nFirst, let's prepare the document images for the model with the help of the `image_processor` from the processor.\nBy default, image processor resizes the images to 224x224, makes sure they have the correct order of color channels,\napplies OCR with tesseract to get words and normalized bounding boxes. In this tutorial, all of these defaults are exactly what we need.\nWrite a function that applies the default image processing to a batch of images and returns the results of OCR.\n\n```py\n>>> image_processor = processor.image_processor\n\n\n>>> def get_ocr_words_and_boxes(examples):\n...     images = [image.convert(\"RGB\") for image in examples[\"image\"]]\n...     encoded_inputs = image_processor(images)\n\n...     examples[\"image\"] = encoded_inputs.pixel_values\n...     examples[\"words\"] = encoded_inputs.words\n...     examples[\"boxes\"] = encoded_inputs.boxes\n\n...     return examples\n```\n\nTo apply this preprocessing to the entire dataset in a fast way, use [`~datasets.Dataset.map`].\n\n```py\n>>> dataset_with_ocr = updated_dataset.map(get_ocr_words_and_boxes, batched=True, batch_size=2)\n```\n\n### Preprocessing text data\n\nOnce we have applied OCR to the images, we need to encode the text part of the dataset to prepare it for the model.\nThis involves converting the words and boxes that we got in the previous step to token-level `input_ids`, `attention_mask`,\n`token_type_ids` and `bbox`. For preprocessing text, we'll need the `tokenizer` from the processor.\n\n```py\n>>> tokenizer = processor.tokenizer\n```\n\nOn top of the preprocessing mentioned above, we also need to add the labels for the model. For `xxxForQuestionAnswering` models\nin \ud83e\udd17 Transformers, the labels consist of the `start_positions` and `end_positions`, indicating which token is at the\nstart and which token is at the end of the answer.\n\nLet's start with that. Define a helper function that can find a sublist (the answer split into words) in a larger list (the words list).\n\nThis function will take two lists as input, `words_list` and `answer_list`. It will then iterate over the `words_list` and check\nif the current word in the `words_list` (words_list[i]) is equal to the first word of answer_list (answer_list[0]) and if\nthe sublist of `words_list` starting from the current word and of the same length as `answer_list` is equal `to answer_list`.\nIf this condition is true, it means that a match has been found, and the function will record the match, its starting index (idx),\nand its ending index (idx + len(answer_list) - 1). If more than one match was found, the function will return only the first one.\nIf no match is found, the function returns (`None`, 0, and 0)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9f316b23-bd3e-42d9-a720-b635fa3c708a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9f316b23-bd3e-42d9-a720-b635fa3c708a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"39618fb6-68bf-485b-936c-89bd83f4658f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"506d558068d4251ed82cbd3df45135a6aa984d9f1c95aad9a79277de941a3f1a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"891aa365-c5fe-4bbe-ad49-574efc9171c3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1be996007826f65ac69e35b683e3a4fd5423b560e465f82e8437dd414e3bf2e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7534, \"end_char_idx\": 10500, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "```py\n>>> dataset_with_ocr = updated_dataset.map(get_ocr_words_and_boxes, batched=True, batch_size=2)\n```\n\n### Preprocessing text data\n\nOnce we have applied OCR to the images, we need to encode the text part of the dataset to prepare it for the model.\nThis involves converting the words and boxes that we got in the previous step to token-level `input_ids`, `attention_mask`,\n`token_type_ids` and `bbox`. For preprocessing text, we'll need the `tokenizer` from the processor.\n\n```py\n>>> tokenizer = processor.tokenizer\n```\n\nOn top of the preprocessing mentioned above, we also need to add the labels for the model. For `xxxForQuestionAnswering` models\nin \ud83e\udd17 Transformers, the labels consist of the `start_positions` and `end_positions`, indicating which token is at the\nstart and which token is at the end of the answer.\n\nLet's start with that. Define a helper function that can find a sublist (the answer split into words) in a larger list (the words list).\n\nThis function will take two lists as input, `words_list` and `answer_list`. It will then iterate over the `words_list` and check\nif the current word in the `words_list` (words_list[i]) is equal to the first word of answer_list (answer_list[0]) and if\nthe sublist of `words_list` starting from the current word and of the same length as `answer_list` is equal `to answer_list`.\nIf this condition is true, it means that a match has been found, and the function will record the match, its starting index (idx),\nand its ending index (idx + len(answer_list) - 1). If more than one match was found, the function will return only the first one.\nIf no match is found, the function returns (`None`, 0, and 0).\n\n```py\n>>> def subfinder(words_list, answer_list):\n...     matches = []\n...     start_indices = []\n...     end_indices = []\n...     for idx, i in enumerate(range(len(words_list))):\n...         if words_list[i] == answer_list[0] and words_list[i : i + len(answer_list)] == answer_list:\n...             matches.append(answer_list)\n...             start_indices.append(idx)\n...             end_indices.append(idx + len(answer_list) - 1)\n...     if matches:\n...         return matches[0], start_indices[0], end_indices[0]\n...     else:\n...         return None, 0, 0\n```\n\nTo illustrate how this function finds the position of the answer, let's use it on an example:\n\n```py\n>>> example = dataset_with_ocr[\"train\"][1]\n>>> words = [word.lower() for word in example[\"words\"]]\n>>> match, word_idx_start, word_idx_end = subfinder(words, example[\"answer\"].lower().split())\n>>> print(\"Question: \", example[\"question\"])\n>>> print(\"Words:\", words)\n>>> print(\"Answer: \", example[\"answer\"])\n>>> print(\"start_index\", word_idx_start)\n>>> print(\"end_index\", word_idx_end)\nQuestion:  Who is in  cc in this letter?\nWords: ['wie', 'baw', 'brown', '&', 'williamson', 'tobacco', 'corporation', 'research', '&', 'development', 'internal', 'correspondence', 'to:', 'r.', 'h.', 'honeycutt', 'ce:', 't.f.', 'riehl', 'from:', '.', 'c.j."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 891aa365-c5fe-4bbe-ad49-574efc9171c3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"891aa365-c5fe-4bbe-ad49-574efc9171c3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9f316b23-bd3e-42d9-a720-b635fa3c708a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e72099d6650ec9fc97dad590d0f3864bca50700bec6225d43fe51b164161ec2a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b1045981-da24-4f84-a244-e8c8a194e0da\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea9c2ad6fe327da3d8086b56381da8f049b2d2abf9ddd19de0e2b8824ebccb45\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9052, \"end_char_idx\": 11606, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "If more than one match was found, the function will return only the first one.\nIf no match is found, the function returns (`None`, 0, and 0).\n\n```py\n>>> def subfinder(words_list, answer_list):\n...     matches = []\n...     start_indices = []\n...     end_indices = []\n...     for idx, i in enumerate(range(len(words_list))):\n...         if words_list[i] == answer_list[0] and words_list[i : i + len(answer_list)] == answer_list:\n...             matches.append(answer_list)\n...             start_indices.append(idx)\n...             end_indices.append(idx + len(answer_list) - 1)\n...     if matches:\n...         return matches[0], start_indices[0], end_indices[0]\n...     else:\n...         return None, 0, 0\n```\n\nTo illustrate how this function finds the position of the answer, let's use it on an example:\n\n```py\n>>> example = dataset_with_ocr[\"train\"][1]\n>>> words = [word.lower() for word in example[\"words\"]]\n>>> match, word_idx_start, word_idx_end = subfinder(words, example[\"answer\"].lower().split())\n>>> print(\"Question: \", example[\"question\"])\n>>> print(\"Words:\", words)\n>>> print(\"Answer: \", example[\"answer\"])\n>>> print(\"start_index\", word_idx_start)\n>>> print(\"end_index\", word_idx_end)\nQuestion:  Who is in  cc in this letter?\nWords: ['wie', 'baw', 'brown', '&', 'williamson', 'tobacco', 'corporation', 'research', '&', 'development', 'internal', 'correspondence', 'to:', 'r.', 'h.', 'honeycutt', 'ce:', 't.f.', 'riehl', 'from:', '.', 'c.j.', 'cook', 'date:', 'may', '8,', '1995', 'subject:', 'review', 'of', 'existing', 'brainstorming', 'ideas/483', 'the', 'major', 'function', 'of', 'the', 'product', 'innovation', 'graup', 'is', 'to', 'develop', 'marketable', 'nove!', 'products', 'that', 'would', 'be', 'profitable', 'to', 'manufacture', 'and', 'sell.', 'novel', 'is', 'defined', 'as:', 'of', 'a', 'new', 'kind,', 'or', 'different', 'from', 'anything', 'seen', 'or', 'known', 'before.', 'innovation', 'is', 'defined', 'as:', 'something', 'new', 'or', 'different', 'introduced;', 'act', 'of', 'innovating;', 'introduction', 'of', 'new', 'things', 'or', 'methods.', 'the', 'products', 'may', 'incorporate', 'the', 'latest', 'technologies,', 'materials', 'and', 'know-how', 'available', 'to', 'give', 'then', 'a', 'unique', 'taste', 'or', 'look.', 'the', 'first', 'task', 'of', 'the', 'product', 'innovation', 'group', 'was', 'to', 'assemble,', 'review', 'and', 'categorize', 'a', 'list', 'of', 'existing', 'brainstorming', 'ideas.', 'ideas', 'were', 'grouped', 'into', 'two', 'major', 'categories', 'labeled', 'appearance', 'and', 'taste/aroma."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b1045981-da24-4f84-a244-e8c8a194e0da
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b1045981-da24-4f84-a244-e8c8a194e0da\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"891aa365-c5fe-4bbe-ad49-574efc9171c3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8ab87b63fed3d835f7970c7b3cb08a176b735062628e597684361dd1ee6c9585\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"49a6feb8-c314-4030-a065-f063efbf2bcb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6c23259061dd04b077ba3316885b24d1c5eb20b10475aadfe7b2a78f363e73f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10492, \"end_char_idx\": 12569, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "', 'c.j.', 'cook', 'date:', 'may', '8,', '1995', 'subject:', 'review', 'of', 'existing', 'brainstorming', 'ideas/483', 'the', 'major', 'function', 'of', 'the', 'product', 'innovation', 'graup', 'is', 'to', 'develop', 'marketable', 'nove!', 'products', 'that', 'would', 'be', 'profitable', 'to', 'manufacture', 'and', 'sell.', 'novel', 'is', 'defined', 'as:', 'of', 'a', 'new', 'kind,', 'or', 'different', 'from', 'anything', 'seen', 'or', 'known', 'before.', 'innovation', 'is', 'defined', 'as:', 'something', 'new', 'or', 'different', 'introduced;', 'act', 'of', 'innovating;', 'introduction', 'of', 'new', 'things', 'or', 'methods.', 'the', 'products', 'may', 'incorporate', 'the', 'latest', 'technologies,', 'materials', 'and', 'know-how', 'available', 'to', 'give', 'then', 'a', 'unique', 'taste', 'or', 'look.', 'the', 'first', 'task', 'of', 'the', 'product', 'innovation', 'group', 'was', 'to', 'assemble,', 'review', 'and', 'categorize', 'a', 'list', 'of', 'existing', 'brainstorming', 'ideas.', 'ideas', 'were', 'grouped', 'into', 'two', 'major', 'categories', 'labeled', 'appearance', 'and', 'taste/aroma.', 'these', 'categories', 'are', 'used', 'for', 'novel', 'products', 'that', 'may', 'differ', 'from', 'a', 'visual', 'and/or', 'taste/aroma', 'point', 'of', 'view', 'compared', 'to', 'canventional', 'cigarettes.', 'other', 'categories', 'include', 'a', 'combination', 'of', 'the', 'above,', 'filters,', 'packaging', 'and', 'brand', 'extensions.', 'appearance', 'this', 'category', 'is', 'used', 'for', 'novel', 'cigarette', 'constructions', 'that', 'yield', 'visually', 'different', 'products', 'with', 'minimal', 'changes', 'in', 'smoke', 'chemistry', 'two', 'cigarettes', 'in', 'cne.', 'emulti-plug', 'te', 'build', 'yaur', 'awn', 'cigarette.', 'eswitchable', 'menthol', 'or', 'non', 'menthol', 'cigarette.', '*cigarettes', 'with', 'interspaced', 'perforations', 'to', 'enable', 'smoker', 'to', 'separate', 'unburned', 'section', 'for', 'future', 'smoking.', '\u00abshort', 'cigarette,', 'tobacco', 'section', '30', 'mm.', '\u00abextremely', 'fast', 'buming', 'cigarette."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 49a6feb8-c314-4030-a065-f063efbf2bcb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"49a6feb8-c314-4030-a065-f063efbf2bcb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b1045981-da24-4f84-a244-e8c8a194e0da\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"74d3177ca4c57ef59428745a8e3b053126181647678349f42aa4daccd9ce7bb5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ff83a6ac-e877-46fd-be94-214eb7fba489\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"253e53b788e21308ff0d22b7e8776240d5b82a2929f38ffb5874994a70b8906d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11492, \"end_char_idx\": 13724, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "', 'ideas', 'were', 'grouped', 'into', 'two', 'major', 'categories', 'labeled', 'appearance', 'and', 'taste/aroma.', 'these', 'categories', 'are', 'used', 'for', 'novel', 'products', 'that', 'may', 'differ', 'from', 'a', 'visual', 'and/or', 'taste/aroma', 'point', 'of', 'view', 'compared', 'to', 'canventional', 'cigarettes.', 'other', 'categories', 'include', 'a', 'combination', 'of', 'the', 'above,', 'filters,', 'packaging', 'and', 'brand', 'extensions.', 'appearance', 'this', 'category', 'is', 'used', 'for', 'novel', 'cigarette', 'constructions', 'that', 'yield', 'visually', 'different', 'products', 'with', 'minimal', 'changes', 'in', 'smoke', 'chemistry', 'two', 'cigarettes', 'in', 'cne.', 'emulti-plug', 'te', 'build', 'yaur', 'awn', 'cigarette.', 'eswitchable', 'menthol', 'or', 'non', 'menthol', 'cigarette.', '*cigarettes', 'with', 'interspaced', 'perforations', 'to', 'enable', 'smoker', 'to', 'separate', 'unburned', 'section', 'for', 'future', 'smoking.', '\u00abshort', 'cigarette,', 'tobacco', 'section', '30', 'mm.', '\u00abextremely', 'fast', 'buming', 'cigarette.', '\u00abnovel', 'cigarette', 'constructions', 'that', 'permit', 'a', 'significant', 'reduction', 'iretobacco', 'weight', 'while', 'maintaining', 'smoking', 'mechanics', 'and', 'visual', 'characteristics.', 'higher', 'basis', 'weight', 'paper:', 'potential', 'reduction', 'in', 'tobacco', 'weight.', '\u00abmore', 'rigid', 'tobacco', 'column;', 'stiffing', 'agent', 'for', 'tobacco;', 'e.g.', 'starch', '*colored', 'tow', 'and', 'cigarette', 'papers;', 'seasonal', 'promotions,', 'e.g.', 'pastel', 'colored', 'cigarettes', 'for', 'easter', 'or', 'in', 'an', 'ebony', 'and', 'ivory', 'brand', 'containing', 'a', 'mixture', 'of', 'all', 'black', '(black', 'paper', 'and', 'tow)', 'and', 'ail', 'white', 'cigarettes.', '499150498']\nAnswer:  T.F. Riehl\nstart_index 17\nend_index 18\n```\n\nOnce examples are encoded, however, they will look like this:\n\n```py\n>>> encoding = tokenizer(example[\"question\"], example[\"words\"], example[\"boxes\"])\n>>> tokenizer.decode(encoding[\"input_ids\"])\n[CLS] who is in cc in this letter? [SEP] wie baw brown & williamson tobacco corporation research & development ...\n```\n\nWe'll need to find the position of the answer in the encoded input."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ff83a6ac-e877-46fd-be94-214eb7fba489
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ff83a6ac-e877-46fd-be94-214eb7fba489\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"49a6feb8-c314-4030-a065-f063efbf2bcb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"445b0700783de8e079d7d1f0f60e15ad6ec36eacaadb511009808ffd7d2f4720\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8c3e84d5-3d47-4532-be37-31fc8aa2ca33\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"54858536e85b1941144a1b1dd7af4ca00dafcc251b84a08e6b6410c9b52b4338\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "', '\u00abextremely', 'fast', 'buming', 'cigarette.', '\u00abnovel', 'cigarette', 'constructions', 'that', 'permit', 'a', 'significant', 'reduction', 'iretobacco', 'weight', 'while', 'maintaining', 'smoking', 'mechanics', 'and', 'visual', 'characteristics.', 'higher', 'basis', 'weight', 'paper:', 'potential', 'reduction', 'in', 'tobacco', 'weight.', '\u00abmore', 'rigid', 'tobacco', 'column;', 'stiffing', 'agent', 'for', 'tobacco;', 'e.g.', 'starch', '*colored', 'tow', 'and', 'cigarette', 'papers;', 'seasonal', 'promotions,', 'e.g.', 'pastel', 'colored', 'cigarettes', 'for', 'easter', 'or', 'in', 'an', 'ebony', 'and', 'ivory', 'brand', 'containing', 'a', 'mixture', 'of', 'all', 'black', '(black', 'paper', 'and', 'tow)', 'and', 'ail', 'white', 'cigarettes.', '499150498']\nAnswer:  T.F. Riehl\nstart_index 17\nend_index 18\n```\n\nOnce examples are encoded, however, they will look like this:\n\n```py\n>>> encoding = tokenizer(example[\"question\"], example[\"words\"], example[\"boxes\"])\n>>> tokenizer.decode(encoding[\"input_ids\"])\n[CLS] who is in cc in this letter? [SEP] wie baw brown & williamson tobacco corporation research & development ...\n```\n\nWe'll need to find the position of the answer in the encoded input.\n* `token_type_ids` tells us which tokens are part of the question, and which ones are part of the document's words.\n* `tokenizer.cls_token_id` will help find the special token at the beginning of the input.\n* `word_ids` will help match the answer found in the original `words` to the same answer in the full encoded input and determine\nthe start/end position of the answer in the encoded input.\n\nWith that in mind, let's create a function to encode a batch of examples in the dataset:\n\n```py\n>>> def encode_dataset(examples, max_length=512):\n.     questions = examples[\"question\"]\n.     words = examples[\"words\"]\n.     boxes = examples[\"boxes\"]\n.     answers = examples[\"answer\"]\n\n.     # encode the batch of examples and initialize the start_positions and end_positions\n.     encoding = tokenizer(questions, words, boxes, max_length=max_length, padding=\"max_length\", truncation=True)\n.     start_positions = []\n.     end_positions = []\n\n.     # loop through the examples in the batch\n.     for i in range(len(questions)):\n.         cls_index = encoding[\"input_ids\"][i].index(tokenizer.cls_token_id)\n\n.         # find the position of the answer in example's words\n.         words_example = [word.lower() for word in words[i]]\n.         answer = answers[i]\n.         match, word_idx_start, word_idx_end = subfinder(words_example, answer.lower().split())\n\n.         if match:\n.             # if match is found, use `token_type_ids` to find where words start in the encoding\n.             token_type_ids = encoding[\"token_type_ids\"][i]\n.             token_start_index = 0\n.             while token_type_ids[token_start_index] != 1:\n.                 token_start_index += 1\n\n."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8c3e84d5-3d47-4532-be37-31fc8aa2ca33
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8c3e84d5-3d47-4532-be37-31fc8aa2ca33\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ff83a6ac-e877-46fd-be94-214eb7fba489\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"49245f42e3ccf604f1e0b3d67d84b3079aed4545152c6946f0708f4ccd7d3a9a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a8b28b90-0385-4c76-81f1-8f9b5eb90d1d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c927672daa6187c26f9de6e8cbcd69667d0de0f3c3bdca2b4b5c19d3cfa5e200\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "* `token_type_ids` tells us which tokens are part of the question, and which ones are part of the document's words.\n* `tokenizer.cls_token_id` will help find the special token at the beginning of the input.\n* `word_ids` will help match the answer found in the original `words` to the same answer in the full encoded input and determine\nthe start/end position of the answer in the encoded input.\n\nWith that in mind, let's create a function to encode a batch of examples in the dataset:\n\n```py\n>>> def encode_dataset(examples, max_length=512):\n.     questions = examples[\"question\"]\n.     words = examples[\"words\"]\n.     boxes = examples[\"boxes\"]\n.     answers = examples[\"answer\"]\n\n.     # encode the batch of examples and initialize the start_positions and end_positions\n.     encoding = tokenizer(questions, words, boxes, max_length=max_length, padding=\"max_length\", truncation=True)\n.     start_positions = []\n.     end_positions = []\n\n.     # loop through the examples in the batch\n.     for i in range(len(questions)):\n.         cls_index = encoding[\"input_ids\"][i].index(tokenizer.cls_token_id)\n\n.         # find the position of the answer in example's words\n.         words_example = [word.lower() for word in words[i]]\n.         answer = answers[i]\n.         match, word_idx_start, word_idx_end = subfinder(words_example, answer.lower().split())\n\n.         if match:\n.             # if match is found, use `token_type_ids` to find where words start in the encoding\n.             token_type_ids = encoding[\"token_type_ids\"][i]\n.             token_start_index = 0\n.             while token_type_ids[token_start_index] != 1:\n.                 token_start_index += 1\n\n.             token_end_index = len(encoding[\"input_ids\"][i]) - 1\n.             while token_type_ids[token_end_index] != 1:\n.                 token_end_index -= 1\n\n.             word_ids = encoding.word_ids(i)[token_start_index : token_end_index + 1]\n.             start_position = cls_index\n.             end_position = cls_index\n\n.             # loop over word_ids and increase `token_start_index` until it matches the answer position in words\n.             # once it matches, save the `token_start_index` as the `start_position` of the answer in the encoding\n.             for id in word_ids:\n.                 if id == word_idx_start:\n.                     start_position = token_start_index\n.                 else:\n.                     token_start_index += 1\n\n.             # similarly loop over `word_ids` starting from the end to find the `end_position` of the answer\n.             for id in word_ids[::-1]:\n.                 if id == word_idx_end:\n.                     end_position = token_end_index\n.                 else:\n.                     token_end_index -= 1\n\n.             start_positions.append(start_position)\n.             end_positions.append(end_position)\n\n.         else:\n.             start_positions.append(cls_index)\n.             end_positions.append(cls_index)\n\n.     encoding[\"image\"] = examples[\"image\"]\n.     encoding[\"start_positions\"] = start_positions\n.     encoding[\"end_positions\"] = end_positions\n\n.     return encoding\n```\n\nNow that we have this preprocessing function, we can encode the entire dataset:\n\n```py\n>>> encoded_train_dataset = dataset_with_ocr[\"train\"].map(\n.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr[\"train\"].column_names\n. )\n>>> encoded_test_dataset = dataset_with_ocr[\"test\"].map(\n.     encode_dataset,"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a8b28b90-0385-4c76-81f1-8f9b5eb90d1d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a8b28b90-0385-4c76-81f1-8f9b5eb90d1d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8c3e84d5-3d47-4532-be37-31fc8aa2ca33\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40ff8c00e45ca710b6202f1d4e8dd710a181554caeeafaaf720d67c14534d2ad\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"651dc2d6-b4fd-4879-a9c6-c6ec0c440db8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5aaf0084b882778a32ed68bbac54f34aee5df2b88aa330d7e5bfd78ae2a700e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "token_end_index = len(encoding[\"input_ids\"][i]) - 1\n.             while token_type_ids[token_end_index] != 1:\n.                 token_end_index -= 1\n\n.             word_ids = encoding.word_ids(i)[token_start_index : token_end_index + 1]\n.             start_position = cls_index\n.             end_position = cls_index\n\n.             # loop over word_ids and increase `token_start_index` until it matches the answer position in words\n.             # once it matches, save the `token_start_index` as the `start_position` of the answer in the encoding\n.             for id in word_ids:\n.                 if id == word_idx_start:\n.                     start_position = token_start_index\n.                 else:\n.                     token_start_index += 1\n\n.             # similarly loop over `word_ids` starting from the end to find the `end_position` of the answer\n.             for id in word_ids[::-1]:\n.                 if id == word_idx_end:\n.                     end_position = token_end_index\n.                 else:\n.                     token_end_index -= 1\n\n.             start_positions.append(start_position)\n.             end_positions.append(end_position)\n\n.         else:\n.             start_positions.append(cls_index)\n.             end_positions.append(cls_index)\n\n.     encoding[\"image\"] = examples[\"image\"]\n.     encoding[\"start_positions\"] = start_positions\n.     encoding[\"end_positions\"] = end_positions\n\n.     return encoding\n```\n\nNow that we have this preprocessing function, we can encode the entire dataset:\n\n```py\n>>> encoded_train_dataset = dataset_with_ocr[\"train\"].map(\n.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr[\"train\"].column_names\n. )\n>>> encoded_test_dataset = dataset_with_ocr[\"test\"].map(\n.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr[\"test\"].column_names\n. )\n```\n\nLet's check what the features of the encoded dataset look like:\n\n```py\n>>> encoded_train_dataset.features\n{'image': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='uint8', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),\n 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),\n 'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),\n 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),\n 'bbox': Sequence(feature=Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), length=-1, id=None),\n 'start_positions': Value(dtype='int64', id=None),\n 'end_positions': Value(dtype='int64', id=None)}\n```\n\n## Evaluation\n\nEvaluation for document question answering requires a significant amount of postprocessing. To avoid taking up too much\nof your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so\nyou're not completely in the dark about your model's performance. Extractive question answering is typically evaluated using F1/exact match.\nIf you'd like to implement it yourself, check out the [Question Answering chapter](https://huggingface.co/course/chapter7/7?fw=pt#postprocessing)\nof the Hugging Face course for inspiration.\n\n## Train\n\nCongratulations! You've successfully navigated the toughest part of this guide and now you are ready to train your own model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 651dc2d6-b4fd-4879-a9c6-c6ec0c440db8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"651dc2d6-b4fd-4879-a9c6-c6ec0c440db8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a8b28b90-0385-4c76-81f1-8f9b5eb90d1d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"22ba5c29400ec391ce9a4faeae571c4030cd38b756a78b1ff71de192c1313c52\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59ae20c8-3b24-412a-a088-95df6846f2dd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2e1b30a09304fb5cc22c2f9e4cdf2223944f6f2965c3355c478a475343182708\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "column_names\n. )\n>>> encoded_test_dataset = dataset_with_ocr[\"test\"].map(\n.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr[\"test\"].column_names\n. )\n```\n\nLet's check what the features of the encoded dataset look like:\n\n```py\n>>> encoded_train_dataset.features\n{'image': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='uint8', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),\n 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),\n 'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),\n 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),\n 'bbox': Sequence(feature=Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), length=-1, id=None),\n 'start_positions': Value(dtype='int64', id=None),\n 'end_positions': Value(dtype='int64', id=None)}\n```\n\n## Evaluation\n\nEvaluation for document question answering requires a significant amount of postprocessing. To avoid taking up too much\nof your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so\nyou're not completely in the dark about your model's performance. Extractive question answering is typically evaluated using F1/exact match.\nIf you'd like to implement it yourself, check out the [Question Answering chapter](https://huggingface.co/course/chapter7/7?fw=pt#postprocessing)\nof the Hugging Face course for inspiration.\n\n## Train\n\nCongratulations! You've successfully navigated the toughest part of this guide and now you are ready to train your own model.\nTraining involves the following steps:\n* Load the model with [`AutoModelForDocumentQuestionAnswering`] using the same checkpoint as in the preprocessing.\n* Define your training hyperparameters in [`TrainingArguments`].\n* Define a function to batch examples together, here the [`DefaultDataCollator`] will do just fine\n* Pass the training arguments to [`Trainer`] along with the model, dataset, and data collator.\n* Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> from transformers import AutoModelForDocumentQuestionAnswering\n\n>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained(model_checkpoint)\n```\n\nIn the [`TrainingArguments`] use `output_dir` to specify where to save your model, and configure hyperparameters as you see fit.\nIf you wish to share your model with the community, set `push_to_hub` to `True` (you must be signed in to Hugging Face to upload your model).\nIn this case the `output_dir` will also be the name of the repo where your model checkpoint will be pushed.\n\n```py\n>>> from transformers import TrainingArguments\n\n>>> # REPLACE THIS WITH YOUR REPO ID\n>>> repo_id = \"MariaK/layoutlmv2-base-uncased_finetuned_docvqa\"\n\n>>> training_args = TrainingArguments(\n...     output_dir=repo_id,\n...     per_device_train_batch_size=4,\n...     num_train_epochs=20,\n...     save_steps=200,\n...     logging_steps=50,\n...     eval_strategy=\"steps\",\n...     learning_rate=5e-5,\n...     save_total_limit=2,\n...     remove_unused_columns=False,\n...     push_to_hub=True,\n... )\n```\n\nDefine a simple data collator to batch examples together."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 59ae20c8-3b24-412a-a088-95df6846f2dd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"59ae20c8-3b24-412a-a088-95df6846f2dd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"651dc2d6-b4fd-4879-a9c6-c6ec0c440db8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"55cab5a5f987f1deb3fc3c95567cce47e47407959d4e64d1915b7f478810e242\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"92f198db-5ef1-4150-b289-0c74de8b476b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"22529ad00d60bfd6d24635cffbca33838f8fe8329dcc8e26d21e8de59a4d3278\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18719, \"end_char_idx\": 21662, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "## Train\n\nCongratulations! You've successfully navigated the toughest part of this guide and now you are ready to train your own model.\nTraining involves the following steps:\n* Load the model with [`AutoModelForDocumentQuestionAnswering`] using the same checkpoint as in the preprocessing.\n* Define your training hyperparameters in [`TrainingArguments`].\n* Define a function to batch examples together, here the [`DefaultDataCollator`] will do just fine\n* Pass the training arguments to [`Trainer`] along with the model, dataset, and data collator.\n* Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> from transformers import AutoModelForDocumentQuestionAnswering\n\n>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained(model_checkpoint)\n```\n\nIn the [`TrainingArguments`] use `output_dir` to specify where to save your model, and configure hyperparameters as you see fit.\nIf you wish to share your model with the community, set `push_to_hub` to `True` (you must be signed in to Hugging Face to upload your model).\nIn this case the `output_dir` will also be the name of the repo where your model checkpoint will be pushed.\n\n```py\n>>> from transformers import TrainingArguments\n\n>>> # REPLACE THIS WITH YOUR REPO ID\n>>> repo_id = \"MariaK/layoutlmv2-base-uncased_finetuned_docvqa\"\n\n>>> training_args = TrainingArguments(\n...     output_dir=repo_id,\n...     per_device_train_batch_size=4,\n...     num_train_epochs=20,\n...     save_steps=200,\n...     logging_steps=50,\n...     eval_strategy=\"steps\",\n...     learning_rate=5e-5,\n...     save_total_limit=2,\n...     remove_unused_columns=False,\n...     push_to_hub=True,\n... )\n```\n\nDefine a simple data collator to batch examples together.\n\n```py\n>>> from transformers import DefaultDataCollator\n\n>>> data_collator = DefaultDataCollator()\n```\n\nFinally, bring everything together, and call [`~Trainer.train`]:\n\n```py\n>>> from transformers import Trainer\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     data_collator=data_collator,\n...     train_dataset=encoded_train_dataset,\n...     eval_dataset=encoded_test_dataset,\n...     tokenizer=processor,\n... )\n\n>>> trainer.train()\n```\n\nTo add the final model to \ud83e\udd17 Hub, create a model card and call `push_to_hub`:\n\n```py\n>>> trainer.create_model_card()\n>>> trainer.push_to_hub()\n```\n\n## Inference\n\nNow that you have finetuned a LayoutLMv2 model, and uploaded it to the \ud83e\udd17 Hub, you can use it for inference. The simplest\nway to try out your finetuned model for inference is to use it in a [`Pipeline`].\n\nLet's take an example:\n```py\n>>> example = dataset[\"test\"][2]\n>>> question = example[\"query\"][\"en\"]\n>>> image = example[\"image\"]\n>>> print(question)\n>>> print(example[\"answers\"])\n'Who is \u2018presiding\u2019 TRRF GENERAL SESSION (PART 1)?'\n['TRRF Vice President', 'lee a. waller']\n```\n\nNext, instantiate a pipeline for\ndocument question answering with your model, and pass the image + question combination to it."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 92f198db-5ef1-4150-b289-0c74de8b476b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"92f198db-5ef1-4150-b289-0c74de8b476b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bf4a080-ad00-4d2d-b673-a7d15d31d851\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"59ae20c8-3b24-412a-a088-95df6846f2dd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/document_question_answering\", \"title\": \"Document Question Answering\", \"tokens\": 5943, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ed4be3b966dfacd1bb06d0e0d2fdaa77b8b0225c42645422d1029ff3f0c37bea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20423, \"end_char_idx\": 23379, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "ref_doc_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "chroma:document": "```py\n>>> from transformers import DefaultDataCollator\n\n>>> data_collator = DefaultDataCollator()\n```\n\nFinally, bring everything together, and call [`~Trainer.train`]:\n\n```py\n>>> from transformers import Trainer\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     data_collator=data_collator,\n...     train_dataset=encoded_train_dataset,\n...     eval_dataset=encoded_test_dataset,\n...     tokenizer=processor,\n... )\n\n>>> trainer.train()\n```\n\nTo add the final model to \ud83e\udd17 Hub, create a model card and call `push_to_hub`:\n\n```py\n>>> trainer.create_model_card()\n>>> trainer.push_to_hub()\n```\n\n## Inference\n\nNow that you have finetuned a LayoutLMv2 model, and uploaded it to the \ud83e\udd17 Hub, you can use it for inference. The simplest\nway to try out your finetuned model for inference is to use it in a [`Pipeline`].\n\nLet's take an example:\n```py\n>>> example = dataset[\"test\"][2]\n>>> question = example[\"query\"][\"en\"]\n>>> image = example[\"image\"]\n>>> print(question)\n>>> print(example[\"answers\"])\n'Who is \u2018presiding\u2019 TRRF GENERAL SESSION (PART 1)?'\n['TRRF Vice President', 'lee a. waller']\n```\n\nNext, instantiate a pipeline for\ndocument question answering with your model, and pass the image + question combination to it.\n\n```py\n>>> from transformers import pipeline\n\n>>> qa_pipeline = pipeline(\"document-question-answering\", model=\"MariaK/layoutlmv2-base-uncased_finetuned_docvqa\")\n>>> qa_pipeline(image, question)\n[{'score': 0.9949808120727539,\n  'answer': 'Lee A. Waller',\n  'start': 55,\n  'end': 57}]\n```\n\nYou can also manually replicate the results of the pipeline if you'd like:\n1. Take an image and a question, prepare them for the model using the processor from your model.\n2. Forward the result or preprocessing through the model.\n3. The model returns `start_logits` and `end_logits`, which indicate which token is at the start of the answer and\nwhich token is at the end of the answer. Both have shape (batch_size, sequence_length).\n4. Take an argmax on the last dimension of both the `start_logits` and `end_logits` to get the predicted `start_idx` and `end_idx`.\n5. Decode the answer with the tokenizer.\n\n```py\n>>> import torch\n>>> from transformers import AutoProcessor\n>>> from transformers import AutoModelForDocumentQuestionAnswering\n\n>>> processor = AutoProcessor.from_pretrained(\"MariaK/layoutlmv2-base-uncased_finetuned_docvqa\")\n>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained(\"MariaK/layoutlmv2-base-uncased_finetuned_docvqa\")\n\n>>> with torch.no_grad():\n...     encoding = processor(image.convert(\"RGB\"), question, return_tensors=\"pt\")\n...     outputs = model(**encoding)\n...     start_logits = outputs.start_logits\n...     end_logits = outputs.end_logits\n...     predicted_start_idx = start_logits.argmax(-1).item()\n...     predicted_end_idx = end_logits.argmax(-1).item()\n\n>>> processor.tokenizer.decode(encoding.input_ids.squeeze()[predicted_start_idx : predicted_end_idx + 1])\n'lee a. waller'\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 10e6c50b-3256-4fce-a430-503e41e29c77
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"10e6c50b-3256-4fce-a430-503e41e29c77\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59e12aa6-e3a9-4979-b7cc-7727cee8386b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"23212da2355ce05c4a828ec946ef7f3ba3f0a16cbfa6a29ddbe3703169d54c2b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3066, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "# Image tasks with IDEFICS\n\n[[open-in-colab]]\n\nWhile individual tasks can be tackled by fine-tuning specialized models, an alternative approach \nthat has recently emerged and gained popularity is to use large models for a diverse set of tasks without fine-tuning. \nFor instance, large language models can handle such NLP tasks as summarization, translation, classification, and more. \nThis approach is no longer limited to a single modality, such as text, and in this guide, we will illustrate how you can \nsolve image-text tasks with a large multimodal model called IDEFICS. \n\n[IDEFICS](../model_doc/idefics) is an open-access vision and language model based on [Flamingo](https://huggingface.co/papers/2204.14198), \na state-of-the-art visual language model initially developed by DeepMind. The model accepts arbitrary sequences of image \nand text inputs and generates coherent text as output. It can answer questions about images, describe visual content, \ncreate stories grounded in multiple images, and so on. IDEFICS comes in two variants - [80 billion parameters](https://huggingface.co/HuggingFaceM4/idefics-80b) \nand [9 billion parameters](https://huggingface.co/HuggingFaceM4/idefics-9b), both of which are available on the \ud83e\udd17 Hub. For each variant, you can also find fine-tuned instructed \nversions of the model adapted for conversational use cases.\n\nThis model is exceptionally versatile and can be used for a wide range of image and multimodal tasks. However, \nbeing a large model means it requires significant computational resources and infrastructure. It is up to you to decide whether \nthis approach suits your use case better than fine-tuning specialized models for each individual task. \n\nIn this guide, you'll learn how to: \n- [Load IDEFICS](#loading-the-model) and [load the quantized version of the model](#quantized-model)\n- Use IDEFICS for: \n  - [Image captioning](#image-captioning)\n  - [Prompted image captioning](#prompted-image-captioning)\n  - [Few-shot prompting](#few-shot-prompting)\n  - [Visual question answering](#visual-question-answering)\n  - [Image classification](#image-classification)\n  - [Image-guided text generation](#image-guided-text-generation)\n- [Run inference in batch mode](#running-inference-in-batch-mode)\n- [Run IDEFICS instruct for conversational use](#idefics-instruct-for-conversational-use)\n\nBefore you begin, make sure you have all the necessary libraries installed. \n\n```bash\npip install -q bitsandbytes sentencepiece accelerate transformers\n```\n\n<Tip>\nTo run the following examples with a non-quantized version of the model checkpoint you will need at least 20GB of GPU memory.\n</Tip>\n\n## Loading the model\n\nLet's start by loading the model's 9 billion parameters checkpoint: \n\n```py\n>>> checkpoint = \"HuggingFaceM4/idefics-9b\"\n```\n\nJust like for other Transformers models, you need to load a processor and the model itself from the checkpoint. \nThe IDEFICS processor wraps a [`LlamaTokenizer`] and IDEFICS image processor into a single processor to take care of \npreparing text and image inputs for the model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 59e12aa6-e3a9-4979-b7cc-7727cee8386b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"59e12aa6-e3a9-4979-b7cc-7727cee8386b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"10e6c50b-3256-4fce-a430-503e41e29c77\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f8860b38887097e25f54e5b9ee05863af376245773043139a77bf4d4176bb075\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"242a285b-537a-46d4-80ca-2276c3dc15cc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0e3a1787316ae602c29724caaa88031d9c8349d04c11dd73e2d51ae20a66c287\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1462, \"end_char_idx\": 4670, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "However, \nbeing a large model means it requires significant computational resources and infrastructure. It is up to you to decide whether \nthis approach suits your use case better than fine-tuning specialized models for each individual task. \n\nIn this guide, you'll learn how to: \n- [Load IDEFICS](#loading-the-model) and [load the quantized version of the model](#quantized-model)\n- Use IDEFICS for: \n  - [Image captioning](#image-captioning)\n  - [Prompted image captioning](#prompted-image-captioning)\n  - [Few-shot prompting](#few-shot-prompting)\n  - [Visual question answering](#visual-question-answering)\n  - [Image classification](#image-classification)\n  - [Image-guided text generation](#image-guided-text-generation)\n- [Run inference in batch mode](#running-inference-in-batch-mode)\n- [Run IDEFICS instruct for conversational use](#idefics-instruct-for-conversational-use)\n\nBefore you begin, make sure you have all the necessary libraries installed. \n\n```bash\npip install -q bitsandbytes sentencepiece accelerate transformers\n```\n\n<Tip>\nTo run the following examples with a non-quantized version of the model checkpoint you will need at least 20GB of GPU memory.\n</Tip>\n\n## Loading the model\n\nLet's start by loading the model's 9 billion parameters checkpoint: \n\n```py\n>>> checkpoint = \"HuggingFaceM4/idefics-9b\"\n```\n\nJust like for other Transformers models, you need to load a processor and the model itself from the checkpoint. \nThe IDEFICS processor wraps a [`LlamaTokenizer`] and IDEFICS image processor into a single processor to take care of \npreparing text and image inputs for the model.\n\n```py\n>>> import torch\n\n>>> from transformers import IdeficsForVisionText2Text, AutoProcessor\n\n>>> processor = AutoProcessor.from_pretrained(checkpoint)\n\n>>> model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map=\"auto\")\n```\n\nSetting `device_map` to `\"auto\"` will automatically determine how to load and store the model weights in the most optimized \nmanner given existing devices.\n\n### Quantized model\n\nIf high-memory GPU availability is an issue, you can load the quantized version of the model. To load the model and the \nprocessor in 4bit precision, pass a `BitsAndBytesConfig` to the `from_pretrained` method and the model will be compressed \non the fly while loading.\n\n```py\n>>> import torch\n>>> from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig\n\n>>> quantization_config = BitsAndBytesConfig(\n...     load_in_4bit=True,\n...     bnb_4bit_compute_dtype=torch.float16,\n... )\n\n>>> processor = AutoProcessor.from_pretrained(checkpoint)\n\n>>> model = IdeficsForVisionText2Text.from_pretrained(\n...     checkpoint,\n...     quantization_config=quantization_config,\n...     device_map=\"auto\"\n... )\n```\n\nNow that you have the model loaded in one of the suggested ways, let's move on to exploring tasks that you can use IDEFICS for.\n\n## Image captioning\nImage captioning is the task of predicting a caption for a given image. A common application is to aid visually impaired \npeople navigate through different situations, for instance, explore image content online. \n\nTo illustrate the task, get an image to be captioned, e.g."}


