# embeddings_queue

created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: df2531b2-6101-4c9a-a782-448c92c507d7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"df2531b2-6101-4c9a-a782-448c92c507d7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f868d315-6a68-4fbf-ac9f-ce16652957a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93f59581f64ec2dad37f83f79ee5529f77dd2d0f54d2b11e2b61b6c6bf686bfd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"99331605-45e9-4d29-8010-5accb00f86c9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1b334b64c753c618cac8b4d6a5560427e8d1cdac3887a60e1c9a8b833de2d554\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2842, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "ref_doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "chroma:document": "# Building custom models\n\nThe \ud83e\udd17 Transformers library is designed to be easily extensible. Every model is fully coded in a given subfolder\nof the repository with no abstraction, so you can easily copy a modeling file and tweak it to your needs.\n\nIf you are writing a brand new model, it might be easier to start from scratch. In this tutorial, we will show you\nhow to write a custom model and its configuration so it can be used inside Transformers, and how you can share it\nwith the community (with the code it relies on) so that anyone can use it, even if it's not present in the \ud83e\udd17\nTransformers library. We'll see how to build upon transformers and extend the framework with your hooks and\ncustom code.\n\nWe will illustrate all of this on a ResNet model, by wrapping the ResNet class of the\n[timm library](https://github.com/rwightman/pytorch-image-models) into a [`PreTrainedModel`].\n\n## Writing a custom configuration\n\nBefore we dive into the model, let's first write its configuration. The configuration of a model is an object that\nwill contain all the necessary information to build the model. As we will see in the next section, the model can only\ntake a `config` to be initialized, so we really need that object to be as complete as possible.\n\n<Tip>\n\nModels in the `transformers` library itself generally follow the convention that they accept a `config` object\nin their `__init__` method, and then pass the whole `config` to sub-layers in the model, rather than breaking the \nconfig object into multiple arguments that are all passed individually to sub-layers. Writing your model in this \nstyle results in simpler code with a clear \"source of truth\" for any hyperparameters, and also makes it easier\nto reuse code from other models in `transformers`.\n\n</Tip>\n\nIn our example, we will take a couple of arguments of the ResNet class that we might want to tweak. Different\nconfigurations will then give us the different types of ResNets that are possible. We then just store those arguments,\nafter checking the validity of a few of them.\n\n```python\nfrom transformers import PretrainedConfig\nfrom typing import List\n\n\nclass ResnetConfig(PretrainedConfig):\n    model_type = \"resnet\"\n\n    def __init__(\n        self,\n        block_type=\"bottleneck\",\n        layers: List[int] = [3, 4, 6, 3],\n        num_classes: int = 1000,\n        input_channels: int = 3,\n        cardinality: int = 1,\n        base_width: int = 64,\n        stem_width: int = 64,\n        stem_type: str = \"\",\n        avg_down: bool = False,\n        **kwargs,\n    ):\n        if block_type not in [\"basic\", \"bottleneck\"]:\n            raise ValueError(f\"`block_type` must be 'basic' or bottleneck', got {block_type}.\")\n        if stem_type not in [\"\", \"deep\", \"deep-tiered\"]:\n            raise ValueError(f\"`stem_type` must be '', 'deep' or 'deep-tiered', got {stem_type}.\")"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 99331605-45e9-4d29-8010-5accb00f86c9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"99331605-45e9-4d29-8010-5accb00f86c9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f868d315-6a68-4fbf-ac9f-ce16652957a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93f59581f64ec2dad37f83f79ee5529f77dd2d0f54d2b11e2b61b6c6bf686bfd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"df2531b2-6101-4c9a-a782-448c92c507d7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6c222ba44d381067551beeff414e0990a938e8a399d2137688f170b852560b74\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7c759946-a767-41d8-a117-8f629526a202\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4e25843629c9596925a099d680f7bd20bdcdb649c5dedad0acdab0788f43559e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2123, \"end_char_idx\": 5270, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "ref_doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "chroma:document": "class ResnetConfig(PretrainedConfig):\n    model_type = \"resnet\"\n\n    def __init__(\n        self,\n        block_type=\"bottleneck\",\n        layers: List[int] = [3, 4, 6, 3],\n        num_classes: int = 1000,\n        input_channels: int = 3,\n        cardinality: int = 1,\n        base_width: int = 64,\n        stem_width: int = 64,\n        stem_type: str = \"\",\n        avg_down: bool = False,\n        **kwargs,\n    ):\n        if block_type not in [\"basic\", \"bottleneck\"]:\n            raise ValueError(f\"`block_type` must be 'basic' or bottleneck', got {block_type}.\")\n        if stem_type not in [\"\", \"deep\", \"deep-tiered\"]:\n            raise ValueError(f\"`stem_type` must be '', 'deep' or 'deep-tiered', got {stem_type}.\")\n\n        self.block_type = block_type\n        self.layers = layers\n        self.num_classes = num_classes\n        self.input_channels = input_channels\n        self.cardinality = cardinality\n        self.base_width = base_width\n        self.stem_width = stem_width\n        self.stem_type = stem_type\n        self.avg_down = avg_down\n        super().__init__(**kwargs)\n```\n\nThe three important things to remember when writing you own configuration are the following:\n- you have to inherit from `PretrainedConfig`,\n- the `__init__` of your `PretrainedConfig` must accept any kwargs,\n- those `kwargs` need to be passed to the superclass `__init__`.\n\nThe inheritance is to make sure you get all the functionality from the \ud83e\udd17 Transformers library, while the two other\nconstraints come from the fact a `PretrainedConfig` has more fields than the ones you are setting. When reloading a\nconfig with the `from_pretrained` method, those fields need to be accepted by your config and then sent to the\nsuperclass.\n\nDefining a `model_type` for your configuration (here `model_type=\"resnet\"`) is not mandatory, unless you want to\nregister your model with the auto classes (see last section).\n\nWith this done, you can easily create and save your configuration like you would do with any other model config of the\nlibrary. Here is how we can create a resnet50d config and save it:\n\n```py\nresnet50d_config = ResnetConfig(block_type=\"bottleneck\", stem_width=32, stem_type=\"deep\", avg_down=True)\nresnet50d_config.save_pretrained(\"custom-resnet\")\n```\n\nThis will save a file named `config.json` inside the folder `custom-resnet`. You can then reload your config with the\n`from_pretrained` method:\n\n```py\nresnet50d_config = ResnetConfig.from_pretrained(\"custom-resnet\")\n```\n\nYou can also use any other method of the [`PretrainedConfig`] class, like [`~PretrainedConfig.push_to_hub`] to\ndirectly upload your config to the Hub.\n\n## Writing a custom model\n\nNow that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that\nextracts the hidden features from a batch of images (like [`BertModel`]) and one that is suitable for image\nclassification (like [`BertForSequenceClassification`]).\n\nAs we mentioned before, we'll only write a loose wrapper of the model to keep it simple for this example. The only\nthing we need to do before writing this class is a map between the block types and actual block classes."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7c759946-a767-41d8-a117-8f629526a202
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7c759946-a767-41d8-a117-8f629526a202\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f868d315-6a68-4fbf-ac9f-ce16652957a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93f59581f64ec2dad37f83f79ee5529f77dd2d0f54d2b11e2b61b6c6bf686bfd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"99331605-45e9-4d29-8010-5accb00f86c9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d720f49f892307432cb9b0668690fbecb9f36d9b9f509e80ba8864fe23a41772\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"057b3393-f93f-422a-b79f-67d3f40150e0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"224883b5d9e07885d738dd4d88feaafee7fa0b56166cd2dbc455738e64695e2e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3703, \"end_char_idx\": 6347, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "ref_doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "chroma:document": "When reloading a\nconfig with the `from_pretrained` method, those fields need to be accepted by your config and then sent to the\nsuperclass.\n\nDefining a `model_type` for your configuration (here `model_type=\"resnet\"`) is not mandatory, unless you want to\nregister your model with the auto classes (see last section).\n\nWith this done, you can easily create and save your configuration like you would do with any other model config of the\nlibrary. Here is how we can create a resnet50d config and save it:\n\n```py\nresnet50d_config = ResnetConfig(block_type=\"bottleneck\", stem_width=32, stem_type=\"deep\", avg_down=True)\nresnet50d_config.save_pretrained(\"custom-resnet\")\n```\n\nThis will save a file named `config.json` inside the folder `custom-resnet`. You can then reload your config with the\n`from_pretrained` method:\n\n```py\nresnet50d_config = ResnetConfig.from_pretrained(\"custom-resnet\")\n```\n\nYou can also use any other method of the [`PretrainedConfig`] class, like [`~PretrainedConfig.push_to_hub`] to\ndirectly upload your config to the Hub.\n\n## Writing a custom model\n\nNow that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that\nextracts the hidden features from a batch of images (like [`BertModel`]) and one that is suitable for image\nclassification (like [`BertForSequenceClassification`]).\n\nAs we mentioned before, we'll only write a loose wrapper of the model to keep it simple for this example. The only\nthing we need to do before writing this class is a map between the block types and actual block classes. Then the\nmodel is defined from the configuration by passing everything to the `ResNet` class:\n\n```py\nfrom transformers import PreTrainedModel\nfrom timm.models.resnet import BasicBlock, Bottleneck, ResNet\nfrom .configuration_resnet import ResnetConfig\n\n\nBLOCK_MAPPING = {\"basic\": BasicBlock, \"bottleneck\": Bottleneck}\n\n\nclass ResnetModel(PreTrainedModel):\n    config_class = ResnetConfig\n\n    def __init__(self, config):\n        super().__init__(config)\n        block_layer = BLOCK_MAPPING[config.block_type]\n        self.model = ResNet(\n            block_layer,\n            config.layers,\n            num_classes=config.num_classes,\n            in_chans=config.input_channels,\n            cardinality=config.cardinality,\n            base_width=config.base_width,\n            stem_width=config.stem_width,\n            stem_type=config.stem_type,\n            avg_down=config.avg_down,\n        )\n\n    def forward(self, tensor):\n        return self.model.forward_features(tensor)\n```\n\nFor the model that will classify images, we just change the forward method:\n\n```py\nimport torch"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 057b3393-f93f-422a-b79f-67d3f40150e0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"057b3393-f93f-422a-b79f-67d3f40150e0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f868d315-6a68-4fbf-ac9f-ce16652957a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93f59581f64ec2dad37f83f79ee5529f77dd2d0f54d2b11e2b61b6c6bf686bfd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7c759946-a767-41d8-a117-8f629526a202\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"045f64d63f6614e610386e8900ebad95c5bab769af2463cb9aa08957cb46a407\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b258e7ce-be23-4332-a522-e8bdb2c28ade\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"702567af40b8191685f370181cc253554298070bba59d092fc7a62c8ebfcae46\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4746, \"end_char_idx\": 8291, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "ref_doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "chroma:document": "## Writing a custom model\n\nNow that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that\nextracts the hidden features from a batch of images (like [`BertModel`]) and one that is suitable for image\nclassification (like [`BertForSequenceClassification`]).\n\nAs we mentioned before, we'll only write a loose wrapper of the model to keep it simple for this example. The only\nthing we need to do before writing this class is a map between the block types and actual block classes. Then the\nmodel is defined from the configuration by passing everything to the `ResNet` class:\n\n```py\nfrom transformers import PreTrainedModel\nfrom timm.models.resnet import BasicBlock, Bottleneck, ResNet\nfrom .configuration_resnet import ResnetConfig\n\n\nBLOCK_MAPPING = {\"basic\": BasicBlock, \"bottleneck\": Bottleneck}\n\n\nclass ResnetModel(PreTrainedModel):\n    config_class = ResnetConfig\n\n    def __init__(self, config):\n        super().__init__(config)\n        block_layer = BLOCK_MAPPING[config.block_type]\n        self.model = ResNet(\n            block_layer,\n            config.layers,\n            num_classes=config.num_classes,\n            in_chans=config.input_channels,\n            cardinality=config.cardinality,\n            base_width=config.base_width,\n            stem_width=config.stem_width,\n            stem_type=config.stem_type,\n            avg_down=config.avg_down,\n        )\n\n    def forward(self, tensor):\n        return self.model.forward_features(tensor)\n```\n\nFor the model that will classify images, we just change the forward method:\n\n```py\nimport torch\n\n\nclass ResnetModelForImageClassification(PreTrainedModel):\n    config_class = ResnetConfig\n\n    def __init__(self, config):\n        super().__init__(config)\n        block_layer = BLOCK_MAPPING[config.block_type]\n        self.model = ResNet(\n            block_layer,\n            config.layers,\n            num_classes=config.num_classes,\n            in_chans=config.input_channels,\n            cardinality=config.cardinality,\n            base_width=config.base_width,\n            stem_width=config.stem_width,\n            stem_type=config.stem_type,\n            avg_down=config.avg_down,\n        )\n\n    def forward(self, tensor, labels=None):\n        logits = self.model(tensor)\n        if labels is not None:\n            loss = torch.nn.cross_entropy(logits, labels)\n            return {\"loss\": loss, \"logits\": logits}\n        return {\"logits\": logits}\n```\n\nIn both cases, notice how we inherit from `PreTrainedModel` and call the superclass initialization with the `config`\n(a bit like when you write a regular `torch.nn.Module`). The line that sets the `config_class` is not mandatory, unless\nyou want to register your model with the auto classes (see last section).\n\n<Tip>\n\nIf your model is very similar to a model inside the library, you can re-use the same configuration as this model.\n\n</Tip>\n\nYou can have your model return anything you want, but returning a dictionary like we did for\n`ResnetModelForImageClassification`, with the loss included when labels are passed, will make your model directly\nusable inside the [`Trainer`] class. Using another output format is fine as long as you are planning on using your own\ntraining loop or another library for training.\n\nNow that we have our model class, let's create one:\n\n```py\nresnet50d = ResnetModelForImageClassification(resnet50d_config)\n```\n\nAgain, you can use any of the methods of [`PreTrainedModel`], like [`~PreTrainedModel.save_pretrained`] or\n[`~PreTrainedModel.push_to_hub`]."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b258e7ce-be23-4332-a522-e8bdb2c28ade
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b258e7ce-be23-4332-a522-e8bdb2c28ade\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f868d315-6a68-4fbf-ac9f-ce16652957a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93f59581f64ec2dad37f83f79ee5529f77dd2d0f54d2b11e2b61b6c6bf686bfd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"057b3393-f93f-422a-b79f-67d3f40150e0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bc1d588a012cf9cede21230e93aa629de754fc8b2243913694211f581a350175\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5f0c5498-e6fe-409e-9d52-84fe2189d73e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"117424f2eb376189b1643b08affd0393634810d9fbe0256ba1afcf05b54ba6d5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7381, \"end_char_idx\": 10738, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "ref_doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "chroma:document": "The line that sets the `config_class` is not mandatory, unless\nyou want to register your model with the auto classes (see last section).\n\n<Tip>\n\nIf your model is very similar to a model inside the library, you can re-use the same configuration as this model.\n\n</Tip>\n\nYou can have your model return anything you want, but returning a dictionary like we did for\n`ResnetModelForImageClassification`, with the loss included when labels are passed, will make your model directly\nusable inside the [`Trainer`] class. Using another output format is fine as long as you are planning on using your own\ntraining loop or another library for training.\n\nNow that we have our model class, let's create one:\n\n```py\nresnet50d = ResnetModelForImageClassification(resnet50d_config)\n```\n\nAgain, you can use any of the methods of [`PreTrainedModel`], like [`~PreTrainedModel.save_pretrained`] or\n[`~PreTrainedModel.push_to_hub`]. We will use the second in the next section, and see how to push the model weights\nwith the code of our model. But first, let's load some pretrained weights inside our model.\n\nIn your own use case, you will probably be training your custom model on your own data. To go fast for this tutorial,\nwe will use the pretrained version of the resnet50d. Since our model is just a wrapper around it, it's going to be\neasy to transfer those weights:\n\n```py\nimport timm\n\npretrained_model = timm.create_model(\"resnet50d\", pretrained=True)\nresnet50d.model.load_state_dict(pretrained_model.state_dict())\n```\n\nNow let's see how to make sure that when we do [`~PreTrainedModel.save_pretrained`] or [`~PreTrainedModel.push_to_hub`], the\ncode of the model is saved.\n\n## Registering a model with custom code to the auto classes\n\nIf you are writing a library that extends \ud83e\udd17 Transformers, you may want to extend the auto classes to include your own\nmodel. This is different from pushing the code to the Hub in the sense that users will need to import your library to\nget the custom models (contrarily to automatically downloading the model code from the Hub).\n\nAs long as your config has a `model_type` attribute that is different from existing model types, and that your model\nclasses have the right `config_class` attributes, you can just add them to the auto classes like this:\n\n```py\nfrom transformers import AutoConfig, AutoModel, AutoModelForImageClassification\n\nAutoConfig.register(\"resnet\", ResnetConfig)\nAutoModel.register(ResnetConfig, ResnetModel)\nAutoModelForImageClassification.register(ResnetConfig, ResnetModelForImageClassification)\n```\n\nNote that the first argument used when registering your custom config to [`AutoConfig`] needs to match the `model_type`\nof your custom config, and the first argument used when registering your custom models to any auto model class needs\nto match the `config_class` of those models.\n\n## Sending the code to the Hub\n\n<Tip warning={true}>\n\nThis API is experimental and may have some slight breaking changes in the next releases.\n\n</Tip>\n\nFirst, make sure your model is fully defined in a `.py` file. It can rely on relative imports to some other files as\nlong as all the files are in the same directory (we don't support submodules for this feature yet). For our example,\nwe'll define a `modeling_resnet.py` file and a `configuration_resnet.py` file in a folder of the current working\ndirectory named `resnet_model`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5f0c5498-e6fe-409e-9d52-84fe2189d73e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5f0c5498-e6fe-409e-9d52-84fe2189d73e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f868d315-6a68-4fbf-ac9f-ce16652957a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93f59581f64ec2dad37f83f79ee5529f77dd2d0f54d2b11e2b61b6c6bf686bfd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b258e7ce-be23-4332-a522-e8bdb2c28ade\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2a6cb14bd89a0dd32e3bf59d557ade7ad9aabefdae42a063ff11bd6fef14ff88\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ef876b28-b1b4-4656-bdc3-10c08a891cfa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ed2c028b3ce8b65cf66d88bd536afe7300e259122596712e4ab91861ccecb80b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9041, \"end_char_idx\": 12465, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "ref_doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "chroma:document": "## Registering a model with custom code to the auto classes\n\nIf you are writing a library that extends \ud83e\udd17 Transformers, you may want to extend the auto classes to include your own\nmodel. This is different from pushing the code to the Hub in the sense that users will need to import your library to\nget the custom models (contrarily to automatically downloading the model code from the Hub).\n\nAs long as your config has a `model_type` attribute that is different from existing model types, and that your model\nclasses have the right `config_class` attributes, you can just add them to the auto classes like this:\n\n```py\nfrom transformers import AutoConfig, AutoModel, AutoModelForImageClassification\n\nAutoConfig.register(\"resnet\", ResnetConfig)\nAutoModel.register(ResnetConfig, ResnetModel)\nAutoModelForImageClassification.register(ResnetConfig, ResnetModelForImageClassification)\n```\n\nNote that the first argument used when registering your custom config to [`AutoConfig`] needs to match the `model_type`\nof your custom config, and the first argument used when registering your custom models to any auto model class needs\nto match the `config_class` of those models.\n\n## Sending the code to the Hub\n\n<Tip warning={true}>\n\nThis API is experimental and may have some slight breaking changes in the next releases.\n\n</Tip>\n\nFirst, make sure your model is fully defined in a `.py` file. It can rely on relative imports to some other files as\nlong as all the files are in the same directory (we don't support submodules for this feature yet). For our example,\nwe'll define a `modeling_resnet.py` file and a `configuration_resnet.py` file in a folder of the current working\ndirectory named `resnet_model`. The configuration file contains the code for `ResnetConfig` and the modeling file\ncontains the code of `ResnetModel` and `ResnetModelForImageClassification`.\n\n```\n.\n\u2514\u2500\u2500 resnet_model\n    \u251c\u2500\u2500 __init__.py\n    \u251c\u2500\u2500 configuration_resnet.py\n    \u2514\u2500\u2500 modeling_resnet.py\n```\n\nThe `__init__.py` can be empty, it's just there so that Python detects `resnet_model` can be use as a module.\n\n<Tip warning={true}>\n\nIf copying a modeling files from the library, you will need to replace all the relative imports at the top of the file\nto import from the `transformers` package.\n\n</Tip>\n\nNote that you can re-use (or subclass) an existing configuration/model.\n\nTo share your model with the community, follow those steps: first import the ResNet model and config from the newly\ncreated files:\n\n```py\nfrom resnet_model.configuration_resnet import ResnetConfig\nfrom resnet_model.modeling_resnet import ResnetModel, ResnetModelForImageClassification\n```\n\nThen you have to tell the library you want to copy the code files of those objects when using the `save_pretrained`\nmethod and properly register them with a given Auto class (especially for models), just run:\n\n```py\nResnetConfig.register_for_auto_class()\nResnetModel.register_for_auto_class(\"AutoModel\")\nResnetModelForImageClassification.register_for_auto_class(\"AutoModelForImageClassification\")\n```\n\nNote that there is no need to specify an auto class for the configuration (there is only one auto class for them,\n[`AutoConfig`]) but it's different for models. Your custom model could be suitable for many different tasks, so you\nhave to specify which one of the auto classes is the correct one for your model.\n\n<Tip>\n\nUse `register_for_auto_class()` if you want the code files to be copied."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ef876b28-b1b4-4656-bdc3-10c08a891cfa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ef876b28-b1b4-4656-bdc3-10c08a891cfa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f868d315-6a68-4fbf-ac9f-ce16652957a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93f59581f64ec2dad37f83f79ee5529f77dd2d0f54d2b11e2b61b6c6bf686bfd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5f0c5498-e6fe-409e-9d52-84fe2189d73e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"de2deb4b685d356e5268e27bf70048dd00473f968f6d7f6fe99905cac8154660\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ef77ca9c-b235-42ec-b9fd-1eec6555357e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0762a7b506e827b292dbf7086cd1992fd1d3acba3c252f22cbcd0b429e11cb15\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10739, \"end_char_idx\": 13922, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "ref_doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "chroma:document": "The configuration file contains the code for `ResnetConfig` and the modeling file\ncontains the code of `ResnetModel` and `ResnetModelForImageClassification`.\n\n```\n.\n\u2514\u2500\u2500 resnet_model\n    \u251c\u2500\u2500 __init__.py\n    \u251c\u2500\u2500 configuration_resnet.py\n    \u2514\u2500\u2500 modeling_resnet.py\n```\n\nThe `__init__.py` can be empty, it's just there so that Python detects `resnet_model` can be use as a module.\n\n<Tip warning={true}>\n\nIf copying a modeling files from the library, you will need to replace all the relative imports at the top of the file\nto import from the `transformers` package.\n\n</Tip>\n\nNote that you can re-use (or subclass) an existing configuration/model.\n\nTo share your model with the community, follow those steps: first import the ResNet model and config from the newly\ncreated files:\n\n```py\nfrom resnet_model.configuration_resnet import ResnetConfig\nfrom resnet_model.modeling_resnet import ResnetModel, ResnetModelForImageClassification\n```\n\nThen you have to tell the library you want to copy the code files of those objects when using the `save_pretrained`\nmethod and properly register them with a given Auto class (especially for models), just run:\n\n```py\nResnetConfig.register_for_auto_class()\nResnetModel.register_for_auto_class(\"AutoModel\")\nResnetModelForImageClassification.register_for_auto_class(\"AutoModelForImageClassification\")\n```\n\nNote that there is no need to specify an auto class for the configuration (there is only one auto class for them,\n[`AutoConfig`]) but it's different for models. Your custom model could be suitable for many different tasks, so you\nhave to specify which one of the auto classes is the correct one for your model.\n\n<Tip>\n\nUse `register_for_auto_class()` if you want the code files to be copied. If you instead prefer to use code on the Hub from another repo, \nyou don't need to call it. In cases where there's more than one auto class, you can modify the `config.json` directly using the \nfollowing structure:\n\n```json\n\"auto_map\": {     \n\t\"AutoConfig\": \"<your-repo-name>--<config-name>\",     \n\t\"AutoModel\": \"<your-repo-name>--<config-name>\",\n\t\"AutoModelFor<Task>\": \"<your-repo-name>--<config-name>\",    \n},\n```\n\n</Tip>\n\nNext, let's create the config and models as we did before:\n\n```py\nresnet50d_config = ResnetConfig(block_type=\"bottleneck\", stem_width=32, stem_type=\"deep\", avg_down=True)\nresnet50d = ResnetModelForImageClassification(resnet50d_config)\n\npretrained_model = timm.create_model(\"resnet50d\", pretrained=True)\nresnet50d.model.load_state_dict(pretrained_model.state_dict())\n```\n\nNow to send the model to the Hub, make sure you are logged in. Either run in your terminal:\n\n```bash\nhuggingface-cli login\n```\n\nor from a notebook:\n\n```py\nfrom huggingface_hub import notebook_login\n\nnotebook_login()\n```\n\nYou can then push to your own namespace (or an organization you are a member of) like this:\n\n```py\nresnet50d.push_to_hub(\"custom-resnet50d\")\n```\n\nOn top of the modeling weights and the configuration in json format, this also copied the modeling and\nconfiguration `.py` files in the folder `custom-resnet50d` and uploaded the result to the Hub. You can check the result\nin this [model repo](https://huggingface.co/sgugger/custom-resnet50d)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ef77ca9c-b235-42ec-b9fd-1eec6555357e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ef77ca9c-b235-42ec-b9fd-1eec6555357e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f868d315-6a68-4fbf-ac9f-ce16652957a0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93f59581f64ec2dad37f83f79ee5529f77dd2d0f54d2b11e2b61b6c6bf686bfd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ef876b28-b1b4-4656-bdc3-10c08a891cfa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/custom_models\", \"title\": \"Building custom models\", \"tokens\": 3680, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5f13ddc534f23442aede2e70914686d1dd91fbd782118c3c33ea7bb7a9c27d44\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12466, \"end_char_idx\": 15285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "ref_doc_id": "f868d315-6a68-4fbf-ac9f-ce16652957a0", "chroma:document": "If you instead prefer to use code on the Hub from another repo, \nyou don't need to call it. In cases where there's more than one auto class, you can modify the `config.json` directly using the \nfollowing structure:\n\n```json\n\"auto_map\": {     \n\t\"AutoConfig\": \"<your-repo-name>--<config-name>\",     \n\t\"AutoModel\": \"<your-repo-name>--<config-name>\",\n\t\"AutoModelFor<Task>\": \"<your-repo-name>--<config-name>\",    \n},\n```\n\n</Tip>\n\nNext, let's create the config and models as we did before:\n\n```py\nresnet50d_config = ResnetConfig(block_type=\"bottleneck\", stem_width=32, stem_type=\"deep\", avg_down=True)\nresnet50d = ResnetModelForImageClassification(resnet50d_config)\n\npretrained_model = timm.create_model(\"resnet50d\", pretrained=True)\nresnet50d.model.load_state_dict(pretrained_model.state_dict())\n```\n\nNow to send the model to the Hub, make sure you are logged in. Either run in your terminal:\n\n```bash\nhuggingface-cli login\n```\n\nor from a notebook:\n\n```py\nfrom huggingface_hub import notebook_login\n\nnotebook_login()\n```\n\nYou can then push to your own namespace (or an organization you are a member of) like this:\n\n```py\nresnet50d.push_to_hub(\"custom-resnet50d\")\n```\n\nOn top of the modeling weights and the configuration in json format, this also copied the modeling and\nconfiguration `.py` files in the folder `custom-resnet50d` and uploaded the result to the Hub. You can check the result\nin this [model repo](https://huggingface.co/sgugger/custom-resnet50d).\n\nSee the [sharing tutorial](model_sharing) for more information on the push to Hub method.\n\n## Using a model with custom code\n\nYou can use any configuration, model or tokenizer with custom code files in its repository with the auto-classes and\nthe `from_pretrained` method. All files and code uploaded to the Hub are scanned for malware (refer to the [Hub security](https://huggingface.co/docs/hub/security#malware-scanning) documentation for more information), but you should still \nreview the model code and author to avoid executing malicious code on your machine. Set `trust_remote_code=True` to use\na model with custom code:\n\n```py\nfrom transformers import AutoModelForImageClassification\n\nmodel = AutoModelForImageClassification.from_pretrained(\"sgugger/custom-resnet50d\", trust_remote_code=True)\n```\n\nIt is also strongly encouraged to pass a commit hash as a `revision` to make sure the author of the models did not\nupdate the code with some malicious new lines (unless you fully trust the authors of the models).\n\n```py\ncommit_hash = \"ed94a7c6247d8aedce4647f00f20de6875b5b292\"\nmodel = AutoModelForImageClassification.from_pretrained(\n    \"sgugger/custom-resnet50d\", trust_remote_code=True, revision=commit_hash\n)\n```\n\nNote that when browsing the commit history of the model repo on the Hub, there is a button to easily copy the commit\nhash of any commit."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 71b37dbd-7419-4053-ad09-ab736c996fe1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"71b37dbd-7419-4053-ad09-ab736c996fe1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09ccc9a3-34fc-4a5d-8114-235707d08fe1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"40c36160-58dc-4608-a6dc-fed65be6d7fe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eee08c127d0da344197c7f0d0d399a0bf975872c44e75bc1bb990ea8f7485437\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2752, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "ref_doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "chroma:document": "# Load pretrained instances with an AutoClass\n\nWith so many different Transformer architectures, it can be challenging to create one for your checkpoint. As a part of \ud83e\udd17 Transformers core philosophy to make the library easy, simple and flexible to use, an `AutoClass` automatically infers and loads the correct architecture from a given checkpoint. The `from_pretrained()` method lets you quickly load a pretrained model for any architecture so you don't have to devote time and resources to train a model from scratch. Producing this type of checkpoint-agnostic code means if your code works for one checkpoint, it will work with another checkpoint - as long as it was trained for a similar task - even if the architecture is different.\n\n<Tip>\n\nRemember, architecture refers to the skeleton of the model and checkpoints are the weights for a given architecture. For example, [BERT](https://huggingface.co/google-bert/bert-base-uncased) is an architecture, while `google-bert/bert-base-uncased` is a checkpoint. Model is a general term that can mean either architecture or checkpoint.\n\n</Tip>\n\nIn this tutorial, learn to:\n\n* Load a pretrained tokenizer.\n* Load a pretrained image processor\n* Load a pretrained feature extractor.\n* Load a pretrained processor.\n* Load a pretrained model.\n* Load a model as a backbone.\n\n## AutoTokenizer\n\nNearly every NLP task begins with a tokenizer. A tokenizer converts your input into a format that can be processed by the model.\n\nLoad a tokenizer with [`AutoTokenizer.from_pretrained`]:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n```\n\nThen tokenize your input as shown below:\n\n```py\n>>> sequence = \"In a hole in the ground there lived a hobbit.\"\n>>> print(tokenizer(sequence))\n{'input_ids': [101, 1999, 1037, 4920, 1999, 1996, 2598, 2045, 2973, 1037, 7570, 10322, 4183, 1012, 102], \n 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \n 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}\n```\n\n## AutoImageProcessor\n\nFor vision tasks, an image processor processes the image into the correct input format.\n\n```py\n>>> from transformers import AutoImageProcessor\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\"google/vit-base-patch16-224\")\n```\n\n## AutoBackbone\n\n<div style=\"text-align: center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stages.png\">\n    <figcaption class=\"mt-2 text-center text-sm text-gray-500\">A Swin backbone with multiple stages for outputting a feature map.</figcaption>\n</div>\n\nThe [`AutoBackbone`] lets you use pretrained models as backbones to get feature maps from different stages of the backbone."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 40c36160-58dc-4608-a6dc-fed65be6d7fe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"40c36160-58dc-4608-a6dc-fed65be6d7fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09ccc9a3-34fc-4a5d-8114-235707d08fe1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"71b37dbd-7419-4053-ad09-ab736c996fe1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e9bdbb56903aa09378be5712c61e971e26290f1b48b21abe72eb8ab3d18d8c75\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"878ca33c-86f7-442c-8a74-6ecc6dab43c2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"28f80ca76a2be400d18f73413702348ada1cccdb895d9037527582d29e9b9a5e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1768, \"end_char_idx\": 4537, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "ref_doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "chroma:document": ">>> print(tokenizer(sequence))\n{'input_ids': [101, 1999, 1037, 4920, 1999, 1996, 2598, 2045, 2973, 1037, 7570, 10322, 4183, 1012, 102], \n 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \n 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}\n```\n\n## AutoImageProcessor\n\nFor vision tasks, an image processor processes the image into the correct input format.\n\n```py\n>>> from transformers import AutoImageProcessor\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\"google/vit-base-patch16-224\")\n```\n\n## AutoBackbone\n\n<div style=\"text-align: center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stages.png\">\n    <figcaption class=\"mt-2 text-center text-sm text-gray-500\">A Swin backbone with multiple stages for outputting a feature map.</figcaption>\n</div>\n\nThe [`AutoBackbone`] lets you use pretrained models as backbones to get feature maps from different stages of the backbone. You should specify one of the following parameters in [`~PretrainedConfig.from_pretrained`]:\n\n* `out_indices` is the index of the layer you'd like to get the feature map from\n* `out_features` is the name of the layer you'd like to get the feature map from\n\nThese parameters can be used interchangeably, but if you use both, make sure they're aligned with each other! If you don't pass any of these parameters, the backbone returns the feature map from the last layer.\n\n<div style=\"text-align: center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stage%201.png\">\n    <figcaption class=\"mt-2 text-center text-sm text-gray-500\">A feature map from the first stage of the backbone. The patch partition refers to the model stem.</figcaption>\n</div>\n\nFor example, in the above diagram, to return the feature map from the first stage of the Swin backbone, you can set `out_indices=(1,)`:\n\n```py\n>>> from transformers import AutoImageProcessor, AutoBackbone\n>>> import torch\n>>> from PIL import Image\n>>> import requests\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> processor = AutoImageProcessor.from_pretrained(\"microsoft/swin-tiny-patch4-window7-224\")\n>>> model = AutoBackbone.from_pretrained(\"microsoft/swin-tiny-patch4-window7-224\", out_indices=(1,))\n\n>>> inputs = processor(image, return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n>>> feature_maps = outputs.feature_maps\n```\n\nNow you can access the `feature_maps` object from the first stage of the backbone:\n\n```py\n>>> list(feature_maps[0].shape)\n[1, 96, 56, 56]\n```\n\n## AutoFeatureExtractor\n\nFor audio tasks, a feature extractor processes the audio signal the correct input format."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 878ca33c-86f7-442c-8a74-6ecc6dab43c2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"878ca33c-86f7-442c-8a74-6ecc6dab43c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09ccc9a3-34fc-4a5d-8114-235707d08fe1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"40c36160-58dc-4608-a6dc-fed65be6d7fe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cb8aae3c4142340de9f8b129748b3cf895d69434a90da83d6851c9ff1b6c9356\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dc69a85c-3b8c-4f12-94b5-ffd3a76e362a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"381199319bccdd20b339d58e6c1dce48138341d80a3e5d9655cf689f789194ff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3120, \"end_char_idx\": 6276, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "ref_doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "chroma:document": "If you don't pass any of these parameters, the backbone returns the feature map from the last layer.\n\n<div style=\"text-align: center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stage%201.png\">\n    <figcaption class=\"mt-2 text-center text-sm text-gray-500\">A feature map from the first stage of the backbone. The patch partition refers to the model stem.</figcaption>\n</div>\n\nFor example, in the above diagram, to return the feature map from the first stage of the Swin backbone, you can set `out_indices=(1,)`:\n\n```py\n>>> from transformers import AutoImageProcessor, AutoBackbone\n>>> import torch\n>>> from PIL import Image\n>>> import requests\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> processor = AutoImageProcessor.from_pretrained(\"microsoft/swin-tiny-patch4-window7-224\")\n>>> model = AutoBackbone.from_pretrained(\"microsoft/swin-tiny-patch4-window7-224\", out_indices=(1,))\n\n>>> inputs = processor(image, return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n>>> feature_maps = outputs.feature_maps\n```\n\nNow you can access the `feature_maps` object from the first stage of the backbone:\n\n```py\n>>> list(feature_maps[0].shape)\n[1, 96, 56, 56]\n```\n\n## AutoFeatureExtractor\n\nFor audio tasks, a feature extractor processes the audio signal the correct input format.\n\nLoad a feature extractor with [`AutoFeatureExtractor.from_pretrained`]:\n\n```py\n>>> from transformers import AutoFeatureExtractor\n\n>>> feature_extractor = AutoFeatureExtractor.from_pretrained(\n...     \"ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition\"\n... )\n```\n\n## AutoProcessor\n\nMultimodal tasks require a processor that combines two types of preprocessing tools. For example, the [LayoutLMV2](model_doc/layoutlmv2) model requires an image processor to handle images and a tokenizer to handle text; a processor combines both of them.\n\nLoad a processor with [`AutoProcessor.from_pretrained`]:\n\n```py\n>>> from transformers import AutoProcessor\n\n>>> processor = AutoProcessor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n```\n\n## AutoModel\n\n<frameworkcontent>\n<pt>\nThe `AutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks). For example, load a model for sequence classification with [`AutoModelForSequenceClassification.from_pretrained`]:\n\n```py\n>>> from transformers import AutoModelForSequenceClassification\n\n>>> model = AutoModelForSequenceClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\nEasily reuse the same checkpoint to load an architecture for a different task:\n\n```py\n>>> from transformers import AutoModelForTokenClassification\n\n>>> model = AutoModelForTokenClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\n<Tip warning={true}>\n\nFor PyTorch models, the `from_pretrained()` method uses `torch.load()` which internally uses `pickle` and is known to be insecure. In general, never load a model that could have come from an untrusted source, or that could have been tampered with."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: dc69a85c-3b8c-4f12-94b5-ffd3a76e362a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dc69a85c-3b8c-4f12-94b5-ffd3a76e362a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09ccc9a3-34fc-4a5d-8114-235707d08fe1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"878ca33c-86f7-442c-8a74-6ecc6dab43c2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"23e56357fb82cc16e3c1f923734aa62b333ecb1fc9ccb2e68062bb5e9068c8d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ddf0ad2-51c2-4579-9237-1bdc9ad8e316\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"54ab0752e326d3f02721787158ef44640b3c6cf1eb3e6d64ded4177f27aeb082\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4539, \"end_char_idx\": 7409, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "ref_doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "chroma:document": "Load a feature extractor with [`AutoFeatureExtractor.from_pretrained`]:\n\n```py\n>>> from transformers import AutoFeatureExtractor\n\n>>> feature_extractor = AutoFeatureExtractor.from_pretrained(\n...     \"ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition\"\n... )\n```\n\n## AutoProcessor\n\nMultimodal tasks require a processor that combines two types of preprocessing tools. For example, the [LayoutLMV2](model_doc/layoutlmv2) model requires an image processor to handle images and a tokenizer to handle text; a processor combines both of them.\n\nLoad a processor with [`AutoProcessor.from_pretrained`]:\n\n```py\n>>> from transformers import AutoProcessor\n\n>>> processor = AutoProcessor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n```\n\n## AutoModel\n\n<frameworkcontent>\n<pt>\nThe `AutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks). For example, load a model for sequence classification with [`AutoModelForSequenceClassification.from_pretrained`]:\n\n```py\n>>> from transformers import AutoModelForSequenceClassification\n\n>>> model = AutoModelForSequenceClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\nEasily reuse the same checkpoint to load an architecture for a different task:\n\n```py\n>>> from transformers import AutoModelForTokenClassification\n\n>>> model = AutoModelForTokenClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\n<Tip warning={true}>\n\nFor PyTorch models, the `from_pretrained()` method uses `torch.load()` which internally uses `pickle` and is known to be insecure. In general, never load a model that could have come from an untrusted source, or that could have been tampered with. This security risk is partially mitigated for public models hosted on the Hugging Face Hub, which are [scanned for malware](https://huggingface.co/docs/hub/security-malware) at each commit. See the [Hub documentation](https://huggingface.co/docs/hub/security) for best practices like [signed commit verification](https://huggingface.co/docs/hub/security-gpg#signing-commits-with-gpg) with GPG.\n\nTensorFlow and Flax checkpoints are not affected, and can be loaded within PyTorch architectures using the `from_tf` and `from_flax` kwargs for the `from_pretrained` method to circumvent this issue.\n\n</Tip>\n\nGenerally, we recommend using the `AutoTokenizer` class and the `AutoModelFor` class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next [tutorial](preprocessing), learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.\n</pt>\n<tf>\nFinally, the `TFAutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1ddf0ad2-51c2-4579-9237-1bdc9ad8e316
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1ddf0ad2-51c2-4579-9237-1bdc9ad8e316\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09ccc9a3-34fc-4a5d-8114-235707d08fe1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dc69a85c-3b8c-4f12-94b5-ffd3a76e362a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/autoclass_tutorial\", \"title\": \"Load pretrained instances with an AutoClass\", \"tokens\": 2150, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"073262c53e3caf9f7ca96d4e9eeac35c16768dbbaae8145135ccb6ba2f865c2f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6160, \"end_char_idx\": 8354, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "ref_doc_id": "09ccc9a3-34fc-4a5d-8114-235707d08fe1", "chroma:document": "In general, never load a model that could have come from an untrusted source, or that could have been tampered with. This security risk is partially mitigated for public models hosted on the Hugging Face Hub, which are [scanned for malware](https://huggingface.co/docs/hub/security-malware) at each commit. See the [Hub documentation](https://huggingface.co/docs/hub/security) for best practices like [signed commit verification](https://huggingface.co/docs/hub/security-gpg#signing-commits-with-gpg) with GPG.\n\nTensorFlow and Flax checkpoints are not affected, and can be loaded within PyTorch architectures using the `from_tf` and `from_flax` kwargs for the `from_pretrained` method to circumvent this issue.\n\n</Tip>\n\nGenerally, we recommend using the `AutoTokenizer` class and the `AutoModelFor` class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next [tutorial](preprocessing), learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.\n</pt>\n<tf>\nFinally, the `TFAutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks). For example, load a model for sequence classification with [`TFAutoModelForSequenceClassification.from_pretrained`]:\n\n```py\n>>> from transformers import TFAutoModelForSequenceClassification\n\n>>> model = TFAutoModelForSequenceClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\nEasily reuse the same checkpoint to load an architecture for a different task:\n\n```py\n>>> from transformers import TFAutoModelForTokenClassification\n\n>>> model = TFAutoModelForTokenClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\nGenerally, we recommend using the `AutoTokenizer` class and the `TFAutoModelFor` class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next [tutorial](preprocessing), learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c017be3f-c47a-4993-a9fb-269cef2ee53f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c017be3f-c47a-4993-a9fb-269cef2ee53f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab9934c4-d8a9-4a9a-8b96-9d593c6614b4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e3f8e93acb5d04a644376f51295677234286b733dcf7411faf693460ef9a0458\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2af926e1-999d-4ac3-ac71-4c7933857919\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4e29a365cf288d80c7dab262266ef79d8dbebbd54f2eec6e1ee4c9927ab70a31\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3101, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "doc_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "ref_doc_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "chroma:document": "# CPU inference\n\nWith some optimizations, it is possible to efficiently run large model inference on a CPU. One of these optimization techniques involves compiling the PyTorch code into an intermediate format for high-performance environments like C++. The other technique fuses multiple operations into one kernel to reduce the overhead of running each operation separately.\n\nYou'll learn how to use [BetterTransformer](https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/) for faster inference, and how to convert your PyTorch code to [TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html). If you're using an Intel CPU, you can also use [graph optimizations](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features.html#graph-optimization) from [Intel Extension for PyTorch](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/index.html) to boost inference speed even more. Finally, learn how to use \ud83e\udd17 Optimum to accelerate inference with ONNX Runtime or OpenVINO (if you're using an Intel CPU).\n\n## BetterTransformer\n\nBetterTransformer accelerates inference with its fastpath (native PyTorch specialized implementation of Transformer functions) execution. The two optimizations in the fastpath execution are:\n\n1. fusion, which combines multiple sequential operations into a single \"kernel\" to reduce the number of computation steps\n2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors\n\nBetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention).\n\n<Tip>\n\nBetterTransformer is not supported for all models. Check this [list](https://huggingface.co/docs/optimum/bettertransformer/overview#supported-models) to see if a model supports BetterTransformer.\n\n</Tip>\n\nBefore you start, make sure you have \ud83e\udd17 Optimum [installed](https://huggingface.co/docs/optimum/installation).\n\nEnable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"bigcode/starcoder\")\nmodel.to_bettertransformer()\n```\n\n## TorchScript\n\nTorchScript is an intermediate PyTorch model representation that can be run in production environments where performance is important. You can train a model in PyTorch and then export it to TorchScript to free the model from Python performance constraints. PyTorch [traces](https://pytorch.org/docs/stable/generated/torch.jit.trace.html) a model to return a [`ScriptFunction`] that is optimized with just-in-time compilation (JIT). Compared to the default eager mode, JIT mode in PyTorch typically yields better performance for inference using optimization techniques like operator fusion.\n\nFor a gentle introduction to TorchScript, see the [Introduction to PyTorch TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) tutorial."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2af926e1-999d-4ac3-ac71-4c7933857919
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2af926e1-999d-4ac3-ac71-4c7933857919\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab9934c4-d8a9-4a9a-8b96-9d593c6614b4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e3f8e93acb5d04a644376f51295677234286b733dcf7411faf693460ef9a0458\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c017be3f-c47a-4993-a9fb-269cef2ee53f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6886338a93a580319ed09db781e0185c0ea938e76339698d6f97fec66ddc4ae9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f472e0e-2cab-49a8-93a5-9abcf9e19fe1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"17a3c268249194838a77a1e6aaf373a827a44c6d8a8bab35da0464c6bd50c7a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1265, \"end_char_idx\": 4457, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "doc_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "ref_doc_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "chroma:document": "The two optimizations in the fastpath execution are:\n\n1. fusion, which combines multiple sequential operations into a single \"kernel\" to reduce the number of computation steps\n2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors\n\nBetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention).\n\n<Tip>\n\nBetterTransformer is not supported for all models. Check this [list](https://huggingface.co/docs/optimum/bettertransformer/overview#supported-models) to see if a model supports BetterTransformer.\n\n</Tip>\n\nBefore you start, make sure you have \ud83e\udd17 Optimum [installed](https://huggingface.co/docs/optimum/installation).\n\nEnable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"bigcode/starcoder\")\nmodel.to_bettertransformer()\n```\n\n## TorchScript\n\nTorchScript is an intermediate PyTorch model representation that can be run in production environments where performance is important. You can train a model in PyTorch and then export it to TorchScript to free the model from Python performance constraints. PyTorch [traces](https://pytorch.org/docs/stable/generated/torch.jit.trace.html) a model to return a [`ScriptFunction`] that is optimized with just-in-time compilation (JIT). Compared to the default eager mode, JIT mode in PyTorch typically yields better performance for inference using optimization techniques like operator fusion.\n\nFor a gentle introduction to TorchScript, see the [Introduction to PyTorch TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) tutorial.\n\nWith the [`Trainer`] class, you can enable JIT mode for CPU inference by setting the `--jit_mode_eval` flag:\n\n```bash\npython run_qa.py \\\n--model_name_or_path csarron/bert-base-uncased-squad-v1 \\\n--dataset_name squad \\\n--do_eval \\\n--max_seq_length 384 \\\n--doc_stride 128 \\\n--output_dir /tmp/ \\\n--no_cuda \\\n--jit_mode_eval\n```\n\n<Tip warning={true}>\n\nFor PyTorch >= 1.14.0, JIT-mode could benefit any model for prediction and evaluation since the dict input is supported in `jit.trace`.\n\nFor PyTorch < 1.14.0, JIT-mode could benefit a model if its forward parameter order matches the tuple input order in `jit.trace`, such as a question-answering model. If the forward parameter order does not match the tuple input order in `jit.trace`, like a text classification model, `jit.trace` will fail and we are capturing this with the exception here to make it fallback. Logging is used to notify users.\n\n</Tip>\n\n## IPEX graph optimization\n\nIntel\u00ae Extension for PyTorch (IPEX) provides further optimizations in JIT mode for Intel CPUs, and we recommend combining it with TorchScript for even faster performance. The IPEX [graph optimization](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/graph_optimization.html) fuses operations like Multi-head attention, Concat Linear, Linear + Add, Linear + Gelu, Add + LayerNorm, and more."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3f472e0e-2cab-49a8-93a5-9abcf9e19fe1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3f472e0e-2cab-49a8-93a5-9abcf9e19fe1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab9934c4-d8a9-4a9a-8b96-9d593c6614b4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e3f8e93acb5d04a644376f51295677234286b733dcf7411faf693460ef9a0458\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2af926e1-999d-4ac3-ac71-4c7933857919\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"529873a962bd4b42fbaa1abb350f17ec150d62ba64966b0a98bbd155cd850fc0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9f2ddb7b-c12d-4da2-b06a-3286946cf8e3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"264dbe468f53d819287f59c7b8637853f250e1a7d7872a587eb70aa3d07a9945\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2929, \"end_char_idx\": 5674, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "doc_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "ref_doc_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "chroma:document": "For a gentle introduction to TorchScript, see the [Introduction to PyTorch TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) tutorial.\n\nWith the [`Trainer`] class, you can enable JIT mode for CPU inference by setting the `--jit_mode_eval` flag:\n\n```bash\npython run_qa.py \\\n--model_name_or_path csarron/bert-base-uncased-squad-v1 \\\n--dataset_name squad \\\n--do_eval \\\n--max_seq_length 384 \\\n--doc_stride 128 \\\n--output_dir /tmp/ \\\n--no_cuda \\\n--jit_mode_eval\n```\n\n<Tip warning={true}>\n\nFor PyTorch >= 1.14.0, JIT-mode could benefit any model for prediction and evaluation since the dict input is supported in `jit.trace`.\n\nFor PyTorch < 1.14.0, JIT-mode could benefit a model if its forward parameter order matches the tuple input order in `jit.trace`, such as a question-answering model. If the forward parameter order does not match the tuple input order in `jit.trace`, like a text classification model, `jit.trace` will fail and we are capturing this with the exception here to make it fallback. Logging is used to notify users.\n\n</Tip>\n\n## IPEX graph optimization\n\nIntel\u00ae Extension for PyTorch (IPEX) provides further optimizations in JIT mode for Intel CPUs, and we recommend combining it with TorchScript for even faster performance. The IPEX [graph optimization](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/graph_optimization.html) fuses operations like Multi-head attention, Concat Linear, Linear + Add, Linear + Gelu, Add + LayerNorm, and more.\n\nTo take advantage of these graph optimizations, make sure you have IPEX [installed](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/installation.html):\n\n```bash\npip install intel_extension_for_pytorch\n```\n\nSet the `--use_ipex` and `--jit_mode_eval` flags in the [`Trainer`] class to enable JIT mode with the graph optimizations:\n\n```bash\npython run_qa.py \\\n--model_name_or_path csarron/bert-base-uncased-squad-v1 \\\n--dataset_name squad \\\n--do_eval \\\n--max_seq_length 384 \\\n--doc_stride 128 \\\n--output_dir /tmp/ \\\n--no_cuda \\\n--use_ipex \\\n--jit_mode_eval\n```\n\n## \ud83e\udd17 Optimum\n\n<Tip>\n\nLearn more details about using ORT with \ud83e\udd17 Optimum in the [Optimum Inference with ONNX Runtime](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models) guide. This section only provides a brief and simple example.\n\n</Tip>\n\nONNX Runtime (ORT) is a model accelerator that runs inference on CPUs by default. ORT is supported by \ud83e\udd17 Optimum which can be used in \ud83e\udd17 Transformers, without making too many changes to your code. You only need to replace the \ud83e\udd17 Transformers `AutoClass` with its equivalent [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and load a checkpoint in the ONNX format."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9f2ddb7b-c12d-4da2-b06a-3286946cf8e3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9f2ddb7b-c12d-4da2-b06a-3286946cf8e3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab9934c4-d8a9-4a9a-8b96-9d593c6614b4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e3f8e93acb5d04a644376f51295677234286b733dcf7411faf693460ef9a0458\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3f472e0e-2cab-49a8-93a5-9abcf9e19fe1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_infer_cpu\", \"title\": \"CPU inference\", \"tokens\": 1709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6c7809a940cbcb0efb7a71915d45077fbec8544a76b7d831ad63c63aa5dc4c0a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4206, \"end_char_idx\": 6711, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "doc_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "ref_doc_id": "ab9934c4-d8a9-4a9a-8b96-9d593c6614b4", "chroma:document": "The IPEX [graph optimization](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/graph_optimization.html) fuses operations like Multi-head attention, Concat Linear, Linear + Add, Linear + Gelu, Add + LayerNorm, and more.\n\nTo take advantage of these graph optimizations, make sure you have IPEX [installed](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/installation.html):\n\n```bash\npip install intel_extension_for_pytorch\n```\n\nSet the `--use_ipex` and `--jit_mode_eval` flags in the [`Trainer`] class to enable JIT mode with the graph optimizations:\n\n```bash\npython run_qa.py \\\n--model_name_or_path csarron/bert-base-uncased-squad-v1 \\\n--dataset_name squad \\\n--do_eval \\\n--max_seq_length 384 \\\n--doc_stride 128 \\\n--output_dir /tmp/ \\\n--no_cuda \\\n--use_ipex \\\n--jit_mode_eval\n```\n\n## \ud83e\udd17 Optimum\n\n<Tip>\n\nLearn more details about using ORT with \ud83e\udd17 Optimum in the [Optimum Inference with ONNX Runtime](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models) guide. This section only provides a brief and simple example.\n\n</Tip>\n\nONNX Runtime (ORT) is a model accelerator that runs inference on CPUs by default. ORT is supported by \ud83e\udd17 Optimum which can be used in \ud83e\udd17 Transformers, without making too many changes to your code. You only need to replace the \ud83e\udd17 Transformers `AutoClass` with its equivalent [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and load a checkpoint in the ONNX format.\n\nFor example, if you're running inference on a question answering task, load the [optimum/roberta-base-squad2](https://huggingface.co/optimum/roberta-base-squad2) checkpoint which contains a `model.onnx` file:\n\n```py\nfrom transformers import AutoTokenizer, pipeline\nfrom optimum.onnxruntime import ORTModelForQuestionAnswering\n\nmodel = ORTModelForQuestionAnswering.from_pretrained(\"optimum/roberta-base-squad2\")\ntokenizer = AutoTokenizer.from_pretrained(\"deepset/roberta-base-squad2\")\n\nonnx_qa = pipeline(\"question-answering\", model=model, tokenizer=tokenizer)\n\nquestion = \"What's my name?\"\ncontext = \"My name is Philipp and I live in Nuremberg.\"\npred = onnx_qa(question, context)\n```\n\nIf you have an Intel CPU, take a look at \ud83e\udd17 [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) which supports a variety of compression techniques (quantization, pruning, knowledge distillation) and tools for converting models to the [OpenVINO](https://huggingface.co/docs/optimum/intel/inference) format for higher performance inference."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 48273042-3201-4abe-9559-02d0f2d6cae4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/bertology", "title": "BERTology", "tokens": 562, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"48273042-3201-4abe-9559-02d0f2d6cae4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/bertology\", \"title\": \"BERTology\", \"tokens\": 562, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"170bc6d3-cf73-4e48-817d-b26b1a0d2399\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/bertology\", \"title\": \"BERTology\", \"tokens\": 562, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10e5a6a2247446e6640196ae886427aacd1f744b4ef227463338fd0cb719316f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1576, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "170bc6d3-cf73-4e48-817d-b26b1a0d2399", "doc_id": "170bc6d3-cf73-4e48-817d-b26b1a0d2399", "ref_doc_id": "170bc6d3-cf73-4e48-817d-b26b1a0d2399", "chroma:document": "# BERTology\n\nThere is a growing field of study concerned with investigating the inner working of large-scale transformers like BERT\n(that some call \"BERTology\"). Some good examples of this field are:\n\n\n- BERT Rediscovers the Classical NLP Pipeline by Ian Tenney, Dipanjan Das, Ellie Pavlick:\n  https://arxiv.org/abs/1905.05950\n- Are Sixteen Heads Really Better than One? by Paul Michel, Omer Levy, Graham Neubig: https://arxiv.org/abs/1905.10650\n- What Does BERT Look At? An Analysis of BERT's Attention by Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D.\n  Manning: https://arxiv.org/abs/1906.04341\n- CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure: https://arxiv.org/abs/2210.04633\n\nIn order to help this new field develop, we have included a few additional features in the BERT/GPT/GPT-2 models to\nhelp people access the inner representations, mainly adapted from the great work of Paul Michel\n(https://arxiv.org/abs/1905.10650):\n\n\n- accessing all the hidden-states of BERT/GPT/GPT-2,\n- accessing all the attention weights for each head of BERT/GPT/GPT-2,\n- retrieving heads output values and gradients to be able to compute head importance score and prune head as explained\n  in https://arxiv.org/abs/1905.10650.\n\nTo help you understand and use these features, we have added a specific example script: [bertology.py](https://github.com/huggingface/transformers/tree/main/examples/research_projects/bertology/run_bertology.py) while extract information and prune a model pre-trained on\nGLUE."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5216f848-985c-41f6-8b29-7e2095a50b31
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"5216f848-985c-41f6-8b29-7e2095a50b31\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d59b591a-de7e-4e36-ad1a-09abfd184d35\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d2699a1a364d3677a2084a1a0bde7a9e246ccda461215d92ed5ac6ef92bb8a82\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2378, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "# Testing\n\n\nLet's take a look at how \ud83e\udd17 Transformers models are tested and how you can write new tests and improve the existing ones.\n\nThere are 2 test suites in the repository:\n\n1. `tests` -- tests for the general API\n2. `examples` -- tests primarily for various applications that aren't part of the API\n\n## How transformers are tested\n\n1. Once a PR is submitted it gets tested with 9 CircleCi jobs. Every new commit to that PR gets retested. These jobs\n   are defined in this [config file](https://github.com/huggingface/transformers/tree/main/.circleci/config.yml), so that if needed you can reproduce the same\n   environment on your machine.\n\n   These CI jobs don't run `@slow` tests.\n\n2. There are 3 jobs run by [github actions](https://github.com/huggingface/transformers/actions):\n\n   - [torch hub integration](https://github.com/huggingface/transformers/tree/main/.github/workflows/github-torch-hub.yml): checks whether torch hub\n     integration works.\n\n   - [self-hosted (push)](https://github.com/huggingface/transformers/tree/main/.github/workflows/self-push.yml): runs fast tests on GPU only on commits on\n     `main`. It only runs if a commit on `main` has updated the code in one of the following folders: `src`,\n     `tests`, `.github` (to prevent running on added model cards, notebooks, etc.)\n\n   - [self-hosted runner](https://github.com/huggingface/transformers/tree/main/.github/workflows/self-scheduled.yml): runs normal and slow tests on GPU in\n     `tests` and `examples`:\n\n```bash\nRUN_SLOW=1 pytest tests/\nRUN_SLOW=1 pytest examples/\n```\n\n   The results can be observed [here](https://github.com/huggingface/transformers/actions).\n\n\n\n## Running tests\n\n\n\n### Choosing which tests to run\n\nThis document goes into many details of how tests can be run. If after reading everything, you need even more details\nyou will find them [here](https://docs.pytest.org/en/latest/usage.html).\n\nHere are some most useful ways of running tests.\n\nRun all:\n\n```console\npytest\n```\n\nor:\n\n```bash\nmake test\n```\n\nNote that the latter is defined as:\n\n```bash\npython -m pytest -n auto --dist=loadfile -s -v ./tests/\n```\n\nwhich tells pytest to:\n\n- run as many test processes as they are CPU cores (which could be too many if you don't have a ton of RAM!)\n- ensure that all tests from the same file will be run by the same test process\n- do not capture output\n- run in verbose mode"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d59b591a-de7e-4e36-ad1a-09abfd184d35
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"d59b591a-de7e-4e36-ad1a-09abfd184d35\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5216f848-985c-41f6-8b29-7e2095a50b31\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f825a77429ea1ac83dc232e8d093d375e4e6c0bb519d1af31dbfe85f18d45d50\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b9233179-22fb-4bfc-bf34-f725260d1a1f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"19444e82403b435ed353821215c57c13d8d52893289bf36176c0c2cddb1a6f45\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1658, \"end_char_idx\": 4733, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "## Running tests\n\n\n\n### Choosing which tests to run\n\nThis document goes into many details of how tests can be run. If after reading everything, you need even more details\nyou will find them [here](https://docs.pytest.org/en/latest/usage.html).\n\nHere are some most useful ways of running tests.\n\nRun all:\n\n```console\npytest\n```\n\nor:\n\n```bash\nmake test\n```\n\nNote that the latter is defined as:\n\n```bash\npython -m pytest -n auto --dist=loadfile -s -v ./tests/\n```\n\nwhich tells pytest to:\n\n- run as many test processes as they are CPU cores (which could be too many if you don't have a ton of RAM!)\n- ensure that all tests from the same file will be run by the same test process\n- do not capture output\n- run in verbose mode\n\n\n\n### Getting the list of all tests\n\nAll tests of the test suite:\n\n```bash\npytest --collect-only -q\n```\n\nAll tests of a given test file:\n\n```bash\npytest tests/test_optimization.py --collect-only -q\n```\n\n### Run a specific test module\n\nTo run an individual test module:\n\n```bash\npytest tests/utils/test_logging.py\n```\n\n### Run specific tests\n\nSince unittest is used inside most of the tests, to run specific subtests you need to know the name of the unittest\nclass containing those tests. For example, it could be:\n\n```bash\npytest tests/test_optimization.py::OptimizationTest::test_adam_w\n```\n\nHere:\n\n- `tests/test_optimization.py` - the file with tests\n- `OptimizationTest` - the name of the class\n- `test_adam_w` - the name of the specific test function\n\nIf the file contains multiple classes, you can choose to run only tests of a given class. For example:\n\n```bash\npytest tests/test_optimization.py::OptimizationTest\n```\n\nwill run all the tests inside that class.\n\nAs mentioned earlier you can see what tests are contained inside the `OptimizationTest` class by running:\n\n```bash\npytest tests/test_optimization.py::OptimizationTest --collect-only -q\n```\n\nYou can run tests by keyword expressions.\n\nTo run only tests whose name contains `adam`:\n\n```bash\npytest -k adam tests/test_optimization.py\n```\n\nLogical `and` and `or` can be used to indicate whether all keywords should match or either. `not` can be used to\nnegate.\n\nTo run all tests except those whose name contains `adam`:\n\n```bash\npytest -k \"not adam\" tests/test_optimization.py\n```\n\nAnd you can combine the two patterns in one:\n\n```bash\npytest -k \"ada and not adam\" tests/test_optimization.py\n```\n\nFor example to run both `test_adafactor` and `test_adam_w` you can use:\n\n```bash\npytest -k \"test_adafactor or test_adam_w\" tests/test_optimization.py\n```\n\nNote that we use `or` here, since we want either of the keywords to match to include both.\n\nIf you want to include only tests that include both patterns, `and` is to be used:\n\n```bash\npytest -k \"test and ada\" tests/test_optimization.py\n```\n\n### Run `accelerate` tests\n\nSometimes you need to run `accelerate` tests on your models. For that you can just add `-m accelerate_tests` to your command, if let's say you want to run these tests on `OPT` run:\n\n```bash\nRUN_SLOW=1 pytest -m accelerate_tests tests/models/opt/test_modeling_opt.py\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b9233179-22fb-4bfc-bf34-f725260d1a1f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b9233179-22fb-4bfc-bf34-f725260d1a1f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d59b591a-de7e-4e36-ad1a-09abfd184d35\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"51b7366a36339b64424bd11022062dc71b7240cad9a72f0fdfd99fdd076fd14e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3122be64-c2e7-4822-9259-ce7dbbb938bb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f974b6090a54b5dbd7073ab95b8f0d8e220bfcb8ace9a39e6607283df322a180\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4736, \"end_char_idx\": 7839, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "### Run documentation tests\n\nIn order to test whether the documentation examples are correct, you should check that the `doctests` are passing.\nAs an example, let's use [`WhisperModel.forward`'s docstring](https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/modeling_whisper.py#L1017-L1035):\n\n```python\nr\"\"\"\nReturns:\n\nExample:\n    ```python\n    >>> import torch\n    >>> from transformers import WhisperModel, WhisperFeatureExtractor\n    >>> from datasets import load_dataset\n\n    >>> model = WhisperModel.from_pretrained(\"openai/whisper-base\")\n    >>> feature_extractor = WhisperFeatureExtractor.from_pretrained(\"openai/whisper-base\")\n    >>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n    >>> inputs = feature_extractor(ds[0][\"audio\"][\"array\"], return_tensors=\"pt\")\n    >>> input_features = inputs.input_features\n    >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id\n    >>> last_hidden_state = model(input_features, decoder_input_ids=decoder_input_ids).last_hidden_state\n    >>> list(last_hidden_state.shape)\n    [1, 2, 512]\n    ```\"\"\"\n\n```\n\nJust run the following line to automatically test every docstring example in the desired file:\n```bash\npytest --doctest-modules <path_to_file_or_dir>\n```\nIf the file has a markdown extention, you should add the `--doctest-glob=\"*.md\"` argument.\n\n### Run only modified tests\n\nYou can run the tests related to the unstaged files or the current branch (according to Git) by using [pytest-picked](https://github.com/anapaulagomes/pytest-picked). This is a great way of quickly testing your changes didn't break\nanything, since it won't run the tests related to files you didn't touch.\n\n```bash\npip install pytest-picked\n```\n\n```bash\npytest --picked\n```\n\nAll tests will be run from files and folders which are modified, but not yet committed.\n\n### Automatically rerun failed tests on source modification\n\n[pytest-xdist](https://github.com/pytest-dev/pytest-xdist) provides a very useful feature of detecting all failed\ntests, and then waiting for you to modify files and continuously re-rerun those failing tests until they pass while you\nfix them. So that you don't need to re start pytest after you made the fix. This is repeated until all tests pass after\nwhich again a full run is performed.\n\n```bash\npip install pytest-xdist\n```\n\nTo enter the mode: `pytest -f` or `pytest --looponfail`\n\nFile changes are detected by looking at `looponfailroots` root directories and all of their contents (recursively).\nIf the default for this value does not work for you, you can change it in your project by setting a configuration\noption in `setup.cfg`:\n\n```ini\n[tool:pytest]\nlooponfailroots = transformers tests\n```\n\nor `pytest.ini`/``tox.ini`` files:\n\n```ini\n[pytest]\nlooponfailroots = transformers tests\n```\n\nThis would lead to only looking for file changes in the respective directories, specified relatively to the ini-file\u2019s\ndirectory.\n\n[pytest-watch](https://github.com/joeyespo/pytest-watch) is an alternative implementation of this functionality."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3122be64-c2e7-4822-9259-ce7dbbb938bb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"3122be64-c2e7-4822-9259-ce7dbbb938bb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b9233179-22fb-4bfc-bf34-f725260d1a1f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"589bdffbf779fd08024327ccc51bba7cd6617c0a534db51ffacaef133a80f561\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eaf14355-366c-42de-8595-732b578e1e8f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"917bb5093723d0630fe5e55dbd8341796a235f3296f05bbec98762e0850b3393\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7842, \"end_char_idx\": 9244, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "### Skip a test module\n\nIf you want to run all test modules, except a few you can exclude them by giving an explicit list of tests to run. For\nexample, to run all except `test_modeling_*.py` tests:\n\n```bash\npytest *ls -1 tests/*py | grep -v test_modeling*\n```\n\n### Clearing state\n\nCI builds and when isolation is important (against speed), cache should be cleared:\n\n```bash\npytest --cache-clear tests\n```\n\n### Running tests in parallel\n\nAs mentioned earlier `make test` runs tests in parallel via `pytest-xdist` plugin (`-n X` argument, e.g. `-n 2`\nto run 2 parallel jobs).\n\n`pytest-xdist`'s `--dist=` option allows one to control how the tests are grouped. `--dist=loadfile` puts the\ntests located in one file onto the same process.\n\nSince the order of executed tests is different and unpredictable, if running the test suite with `pytest-xdist`\nproduces failures (meaning we have some undetected coupled tests), use [pytest-replay](https://github.com/ESSS/pytest-replay) to replay the tests in the same order, which should help with then somehow\nreducing that failing sequence to a minimum.\n\n### Test order and repetition\n\nIt's good to repeat the tests several times, in sequence, randomly, or in sets, to detect any potential\ninter-dependency and state-related bugs (tear down). And the straightforward multiple repetition is just good to detect\nsome problems that get uncovered by randomness of DL."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: eaf14355-366c-42de-8595-732b578e1e8f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"eaf14355-366c-42de-8595-732b578e1e8f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3122be64-c2e7-4822-9259-ce7dbbb938bb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"ca714bab12377e05500bd93122739762e2cbced1304abb7e088377b46fd86848\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8cea4289-ff02-4739-ae10-fd36863726da\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7a00815b5ed9a5e98bbbdd173f6c73689aad44eea906d406e7b59362c56f07de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7842, \"end_char_idx\": 11942, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "### Skip a test module\n\nIf you want to run all test modules, except a few you can exclude them by giving an explicit list of tests to run. For\nexample, to run all except `test_modeling_*.py` tests:\n\n```bash\npytest *ls -1 tests/*py | grep -v test_modeling*\n```\n\n### Clearing state\n\nCI builds and when isolation is important (against speed), cache should be cleared:\n\n```bash\npytest --cache-clear tests\n```\n\n### Running tests in parallel\n\nAs mentioned earlier `make test` runs tests in parallel via `pytest-xdist` plugin (`-n X` argument, e.g. `-n 2`\nto run 2 parallel jobs).\n\n`pytest-xdist`'s `--dist=` option allows one to control how the tests are grouped. `--dist=loadfile` puts the\ntests located in one file onto the same process.\n\nSince the order of executed tests is different and unpredictable, if running the test suite with `pytest-xdist`\nproduces failures (meaning we have some undetected coupled tests), use [pytest-replay](https://github.com/ESSS/pytest-replay) to replay the tests in the same order, which should help with then somehow\nreducing that failing sequence to a minimum.\n\n### Test order and repetition\n\nIt's good to repeat the tests several times, in sequence, randomly, or in sets, to detect any potential\ninter-dependency and state-related bugs (tear down). And the straightforward multiple repetition is just good to detect\nsome problems that get uncovered by randomness of DL.\n\n\n#### Repeat tests\n\n- [pytest-flakefinder](https://github.com/dropbox/pytest-flakefinder):\n\n```bash\npip install pytest-flakefinder\n```\n\nAnd then run every test multiple times (50 by default):\n\n```bash\npytest --flake-finder --flake-runs=5 tests/test_failing_test.py\n```\n\n<Tip>\n\nThis plugin doesn't work with `-n` flag from `pytest-xdist`.\n\n</Tip>\n\n<Tip>\n\nThere is another plugin `pytest-repeat`, but it doesn't work with `unittest`.\n\n</Tip>\n\n#### Run tests in a random order\n\n```bash\npip install pytest-random-order\n```\n\nImportant: the presence of `pytest-random-order` will automatically randomize tests, no configuration change or\ncommand line options is required.\n\nAs explained earlier this allows detection of coupled tests - where one test's state affects the state of another. When\n`pytest-random-order` is installed it will print the random seed it used for that session, e.g:\n\n```bash\npytest tests\n[...]\nUsing --random-order-bucket=module\nUsing --random-order-seed=573663\n```\n\nSo that if the given particular sequence fails, you can reproduce it by adding that exact seed, e.g.:\n\n```bash\npytest --random-order-seed=573663\n[...]\nUsing --random-order-bucket=module\nUsing --random-order-seed=573663\n```\n\nIt will only reproduce the exact order if you use the exact same list of tests (or no list at all). Once you start to\nmanually narrowing down the list you can no longer rely on the seed, but have to list them manually in the exact order\nthey failed and tell pytest to not randomize them instead using `--random-order-bucket=none`, e.g.:\n\n```bash\npytest --random-order-bucket=none tests/test_a.py tests/test_c.py tests/test_b.py\n```\n\nTo disable the shuffling for all tests:\n\n```bash\npytest --random-order-bucket=none\n```\n\nBy default `--random-order-bucket=module` is implied, which will shuffle the files on the module levels. It can also\nshuffle on `class`, `package`, `global` and `none` levels. For the complete details please see its\n[documentation](https://github.com/jbasko/pytest-random-order).\n\nAnother randomization alternative is: [`pytest-randomly`](https://github.com/pytest-dev/pytest-randomly). This\nmodule has a very similar functionality/interface, but it doesn't have the bucket modes available in\n`pytest-random-order`. It has the same problem of imposing itself once installed.\n\n### Look and feel variations\n\n#### pytest-sugar\n\n[pytest-sugar](https://github.com/Frozenball/pytest-sugar) is a plugin that improves the look-n-feel, adds a\nprogressbar, and show tests that fail and the assert instantly. It gets activated automatically upon installation.\n\n```bash\npip install pytest-sugar\n```\n\nTo run tests without it, run:\n\n```bash\npytest -p no:sugar\n```\n\nor uninstall it."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8cea4289-ff02-4739-ae10-fd36863726da
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"8cea4289-ff02-4739-ae10-fd36863726da\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"eaf14355-366c-42de-8595-732b578e1e8f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8b91d577342f0d0f60b63e360c98406c55572ef9a451902cbde4414427c5bcd5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0cdb5dd2-e544-4660-8887-875fb915143a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"24f2914553df558d2923be8e790b04a4e9faf48c08ad450511382f210b06682f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11946, \"end_char_idx\": 14908, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "#### Report each sub-test name and its progress\n\nFor a single or a group of tests via `pytest` (after `pip install pytest-pspec`):\n\n```bash\npytest --pspec tests/test_optimization.py\n```\n\n#### Instantly shows failed tests\n\n[pytest-instafail](https://github.com/pytest-dev/pytest-instafail) shows failures and errors instantly instead of\nwaiting until the end of test session.\n\n```bash\npip install pytest-instafail\n```\n\n```bash\npytest --instafail\n```\n\n### To GPU or not to GPU\n\nOn a GPU-enabled setup, to test in CPU-only mode add `CUDA_VISIBLE_DEVICES=\"\"`:\n\n```bash\nCUDA_VISIBLE_DEVICES=\"\" pytest tests/utils/test_logging.py\n```\n\nor if you have multiple gpus, you can specify which one is to be used by `pytest`. For example, to use only the\nsecond gpu if you have gpus `0` and `1`, you can run:\n\n```bash\nCUDA_VISIBLE_DEVICES=\"1\" pytest tests/utils/test_logging.py\n```\n\nThis is handy when you want to run different tasks on different GPUs.\n\nSome tests must be run on CPU-only, others on either CPU or GPU or TPU, yet others on multiple-GPUs. The following skip\ndecorators are used to set the requirements of tests CPU/GPU/TPU-wise:\n\n- `require_torch` - this test will run only under torch\n- `require_torch_gpu` - as `require_torch` plus requires at least 1 GPU\n- `require_torch_multi_gpu` - as `require_torch` plus requires at least 2 GPUs\n- `require_torch_non_multi_gpu` - as `require_torch` plus requires 0 or 1 GPUs\n- `require_torch_up_to_2_gpus` - as `require_torch` plus requires 0 or 1 or 2 GPUs\n- `require_torch_xla` - as `require_torch` plus requires at least 1 TPU\n\nLet's depict the GPU requirements in the following table:\n\n\n| n gpus | decorator                      |\n|--------|--------------------------------|\n| `>= 0` | `@require_torch`               |\n| `>= 1` | `@require_torch_gpu`           |\n| `>= 2` | `@require_torch_multi_gpu`     |\n| `< 2`  | `@require_torch_non_multi_gpu` |\n| `< 3`  | `@require_torch_up_to_2_gpus`  |\n\n\nFor example, here is a test that must be run only when there are 2 or more GPUs available and pytorch is installed:\n\n```python no-style\n@require_torch_multi_gpu\ndef test_example_with_multi_gpu():\n```\n\nIf a test requires `tensorflow` use the `require_tf` decorator. For example:\n\n```python no-style\n@require_tf\ndef test_tf_thing_with_tensorflow():\n```\n\nThese decorators can be stacked. For example, if a test is slow and requires at least one GPU under pytorch, here is\nhow to set it up:\n\n```python no-style\n@require_torch_gpu\n@slow\ndef test_example_slow_on_gpu():\n```\n\nSome decorators like `@parametrized` rewrite test names, therefore `@require_*` skip decorators have to be listed\nlast for them to work correctly. Here is an example of the correct usage:\n\n```python no-style\n@parameterized.expand(...)\n@require_torch_multi_gpu\ndef test_integration_foo():\n```\n\nThis order problem doesn't exist with `@pytest.mark.parametrize`, you can put it first or last and it will still\nwork. But it only works with non-unittests."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0cdb5dd2-e544-4660-8887-875fb915143a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"0cdb5dd2-e544-4660-8887-875fb915143a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8cea4289-ff02-4739-ae10-fd36863726da\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"403f657a2a924320a9eb93dab46486345fba9df1be9a0ad01bdf392542c1548f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ab39dde9-2391-4c96-b3f8-78990ea70fd8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5c029f3134ee9441daac81eecd7c004fddadc9dff8b502bd6f64118c78f104e9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13580, \"end_char_idx\": 16612, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "| n gpus | decorator                      |\n|--------|--------------------------------|\n| `>= 0` | `@require_torch`               |\n| `>= 1` | `@require_torch_gpu`           |\n| `>= 2` | `@require_torch_multi_gpu`     |\n| `< 2`  | `@require_torch_non_multi_gpu` |\n| `< 3`  | `@require_torch_up_to_2_gpus`  |\n\n\nFor example, here is a test that must be run only when there are 2 or more GPUs available and pytorch is installed:\n\n```python no-style\n@require_torch_multi_gpu\ndef test_example_with_multi_gpu():\n```\n\nIf a test requires `tensorflow` use the `require_tf` decorator. For example:\n\n```python no-style\n@require_tf\ndef test_tf_thing_with_tensorflow():\n```\n\nThese decorators can be stacked. For example, if a test is slow and requires at least one GPU under pytorch, here is\nhow to set it up:\n\n```python no-style\n@require_torch_gpu\n@slow\ndef test_example_slow_on_gpu():\n```\n\nSome decorators like `@parametrized` rewrite test names, therefore `@require_*` skip decorators have to be listed\nlast for them to work correctly. Here is an example of the correct usage:\n\n```python no-style\n@parameterized.expand(...)\n@require_torch_multi_gpu\ndef test_integration_foo():\n```\n\nThis order problem doesn't exist with `@pytest.mark.parametrize`, you can put it first or last and it will still\nwork. But it only works with non-unittests.\n\nInside tests:\n\n- How many GPUs are available:\n\n```python\nfrom transformers.testing_utils import get_gpu_count\n\nn_gpu = get_gpu_count()  # works with torch and tf\n```\n\n### Testing with a specific PyTorch backend or device\n\nTo run the test suite on a specific torch device add `TRANSFORMERS_TEST_DEVICE=\"$device\"` where `$device` is the target backend. For example, to test on CPU only:\n\n```bash\nTRANSFORMERS_TEST_DEVICE=\"cpu\" pytest tests/utils/test_logging.py\n```\n\nThis variable is useful for testing custom or less common PyTorch backends such as `mps`, `xpu` or `npu`. It can also be used to achieve the same effect as `CUDA_VISIBLE_DEVICES` by targeting specific GPUs or testing in CPU-only mode.\n\nCertain devices will require an additional import after importing `torch` for the first time. This can be specified using the environment variable `TRANSFORMERS_TEST_BACKEND`:\n\n```bash\nTRANSFORMERS_TEST_BACKEND=\"torch_npu\" pytest tests/utils/test_logging.py\n```\nAlternative backends may also require the replacement of device-specific functions. For example `torch.cuda.manual_seed` may need to be replaced with a device-specific seed setter like `torch.npu.manual_seed` or `torch.xpu.manual_seed` to correctly set a random seed on the device. To specify a new backend with backend-specific device functions when running the test suite, create a Python device specification file `spec.py` in the format:\n\n```python\nimport torch\nimport torch_npu # for xpu, replace it with `import intel_extension_for_pytorch`\n# !! Further additional imports can be added here !!\n\n# Specify the device name (eg. 'cuda', 'cpu', 'npu', 'xpu', 'mps')\nDEVICE_NAME = 'npu'\n\n# Specify device-specific backends to dispatch to."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ab39dde9-2391-4c96-b3f8-78990ea70fd8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ab39dde9-2391-4c96-b3f8-78990ea70fd8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0cdb5dd2-e544-4660-8887-875fb915143a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"e42b58653191f2fd3031e28ced089a484562c268dbf347a22c74218010d2cde6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cbac188d-4f2b-4dbd-9de8-18a230f4ba25\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2a451f1fb0ffa1308309626b131e0faf8fed8ed244dea65a4d4670f8ca61da26\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14910, \"end_char_idx\": 18178, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "Inside tests:\n\n- How many GPUs are available:\n\n```python\nfrom transformers.testing_utils import get_gpu_count\n\nn_gpu = get_gpu_count()  # works with torch and tf\n```\n\n### Testing with a specific PyTorch backend or device\n\nTo run the test suite on a specific torch device add `TRANSFORMERS_TEST_DEVICE=\"$device\"` where `$device` is the target backend. For example, to test on CPU only:\n\n```bash\nTRANSFORMERS_TEST_DEVICE=\"cpu\" pytest tests/utils/test_logging.py\n```\n\nThis variable is useful for testing custom or less common PyTorch backends such as `mps`, `xpu` or `npu`. It can also be used to achieve the same effect as `CUDA_VISIBLE_DEVICES` by targeting specific GPUs or testing in CPU-only mode.\n\nCertain devices will require an additional import after importing `torch` for the first time. This can be specified using the environment variable `TRANSFORMERS_TEST_BACKEND`:\n\n```bash\nTRANSFORMERS_TEST_BACKEND=\"torch_npu\" pytest tests/utils/test_logging.py\n```\nAlternative backends may also require the replacement of device-specific functions. For example `torch.cuda.manual_seed` may need to be replaced with a device-specific seed setter like `torch.npu.manual_seed` or `torch.xpu.manual_seed` to correctly set a random seed on the device. To specify a new backend with backend-specific device functions when running the test suite, create a Python device specification file `spec.py` in the format:\n\n```python\nimport torch\nimport torch_npu # for xpu, replace it with `import intel_extension_for_pytorch`\n# !! Further additional imports can be added here !!\n\n# Specify the device name (eg. 'cuda', 'cpu', 'npu', 'xpu', 'mps')\nDEVICE_NAME = 'npu'\n\n# Specify device-specific backends to dispatch to.\n# If not specified, will fallback to 'default' in 'testing_utils.py`\nMANUAL_SEED_FN = torch.npu.manual_seed\nEMPTY_CACHE_FN = torch.npu.empty_cache\nDEVICE_COUNT_FN = torch.npu.device_count\n```\nThis format also allows for specification of any additional imports required. To use this file to replace equivalent methods in the test suite, set the environment variable `TRANSFORMERS_TEST_DEVICE_SPEC` to the path of the spec file, e.g. `TRANSFORMERS_TEST_DEVICE_SPEC=spec.py`.\n\nCurrently, only `MANUAL_SEED_FN`, `EMPTY_CACHE_FN` and `DEVICE_COUNT_FN` are supported for device-specific dispatch.\n\n### Distributed training\n\n`pytest` can't deal with distributed training directly. If this is attempted - the sub-processes don't do the right\nthing and end up thinking they are `pytest` and start running the test suite in loops. It works, however, if one\nspawns a normal process that then spawns off multiple workers and manages the IO pipes.\n\nHere are some tests that use it:\n\n- [test_trainer_distributed.py](https://github.com/huggingface/transformers/tree/main/tests/trainer/test_trainer_distributed.py)\n- [test_deepspeed.py](https://github.com/huggingface/transformers/tree/main/tests/deepspeed/test_deepspeed.py)\n\nTo jump right into the execution point, search for the `execute_subprocess_async` call in those tests.\n\nYou will need at least 2 GPUs to see these tests in action:\n\n```bash\nCUDA_VISIBLE_DEVICES=0,1 RUN_SLOW=1 pytest -sv tests/test_trainer_distributed.py\n```\n\n### Output capture\n\nDuring test execution any output sent to `stdout` and `stderr` is captured."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: cbac188d-4f2b-4dbd-9de8-18a230f4ba25
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"cbac188d-4f2b-4dbd-9de8-18a230f4ba25\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ab39dde9-2391-4c96-b3f8-78990ea70fd8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"adc1e4d830ac506dcd9933fa5f3ec496185216a329d0cd46dc243b606a3ba79e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1c1ff302-bbd4-4b30-a285-559358028aec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0af4d5c556c3125733e710f7efee3f543a79d1783a9ee4190ae684f181d7d7bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16613, \"end_char_idx\": 19893, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "# If not specified, will fallback to 'default' in 'testing_utils.py`\nMANUAL_SEED_FN = torch.npu.manual_seed\nEMPTY_CACHE_FN = torch.npu.empty_cache\nDEVICE_COUNT_FN = torch.npu.device_count\n```\nThis format also allows for specification of any additional imports required. To use this file to replace equivalent methods in the test suite, set the environment variable `TRANSFORMERS_TEST_DEVICE_SPEC` to the path of the spec file, e.g. `TRANSFORMERS_TEST_DEVICE_SPEC=spec.py`.\n\nCurrently, only `MANUAL_SEED_FN`, `EMPTY_CACHE_FN` and `DEVICE_COUNT_FN` are supported for device-specific dispatch.\n\n### Distributed training\n\n`pytest` can't deal with distributed training directly. If this is attempted - the sub-processes don't do the right\nthing and end up thinking they are `pytest` and start running the test suite in loops. It works, however, if one\nspawns a normal process that then spawns off multiple workers and manages the IO pipes.\n\nHere are some tests that use it:\n\n- [test_trainer_distributed.py](https://github.com/huggingface/transformers/tree/main/tests/trainer/test_trainer_distributed.py)\n- [test_deepspeed.py](https://github.com/huggingface/transformers/tree/main/tests/deepspeed/test_deepspeed.py)\n\nTo jump right into the execution point, search for the `execute_subprocess_async` call in those tests.\n\nYou will need at least 2 GPUs to see these tests in action:\n\n```bash\nCUDA_VISIBLE_DEVICES=0,1 RUN_SLOW=1 pytest -sv tests/test_trainer_distributed.py\n```\n\n### Output capture\n\nDuring test execution any output sent to `stdout` and `stderr` is captured. If a test or a setup method fails, its\naccording captured output will usually be shown along with the failure traceback.\n\nTo disable output capturing and to get the `stdout` and `stderr` normally, use `-s` or `--capture=no`:\n\n```bash\npytest -s tests/utils/test_logging.py\n```\n\nTo send test results to JUnit format output:\n\n```bash\npytest tests --junitxml=result.xml\n```\n\n### Color control\n\nTo have no color (e.g., yellow on white background is not readable):\n\n```bash\npytest --color=no tests/utils/test_logging.py\n```\n\n### Sending test report to online pastebin service\n\nCreating a URL for each test failure:\n\n```bash\npytest --pastebin=failed tests/utils/test_logging.py\n```\n\nThis will submit test run information to a remote Paste service and provide a URL for each failure. You may select\ntests as usual or add for example -x if you only want to send one particular failure.\n\nCreating a URL for a whole test session log:\n\n```bash\npytest --pastebin=all tests/utils/test_logging.py\n```\n\n## Writing tests\n\n\ud83e\udd17 transformers tests are based on `unittest`, but run by `pytest`, so most of the time features from both systems\ncan be used.\n\nYou can read [here](https://docs.pytest.org/en/stable/unittest.html) which features are supported, but the important\nthing to remember is that most `pytest` fixtures don't work. Neither parametrization, but we use the module\n`parameterized` that works in a similar way.\n\n\n### Parametrization\n\nOften, there is a need to run the same test multiple times, but with different arguments. It could be done from within\nthe test, but then there is no way of running that test for just one set of arguments.\n\n```python\n# test_this1.py\nimport unittest\nfrom parameterized import parameterized"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1c1ff302-bbd4-4b30-a285-559358028aec
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"1c1ff302-bbd4-4b30-a285-559358028aec\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cbac188d-4f2b-4dbd-9de8-18a230f4ba25\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f35d3bab9f974f0b790d89600f02d82da7360661547a3af2287ce217ef3ae145\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"74f59c67-d4f1-47e6-9e58-13fa8fa7b7ba\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d9b43fd7e002a4234d0ca023a2dbd2e534dfbe13b6e5b3af3fee55d81ab82fd6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18179, \"end_char_idx\": 21660, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "If a test or a setup method fails, its\naccording captured output will usually be shown along with the failure traceback.\n\nTo disable output capturing and to get the `stdout` and `stderr` normally, use `-s` or `--capture=no`:\n\n```bash\npytest -s tests/utils/test_logging.py\n```\n\nTo send test results to JUnit format output:\n\n```bash\npytest tests --junitxml=result.xml\n```\n\n### Color control\n\nTo have no color (e.g., yellow on white background is not readable):\n\n```bash\npytest --color=no tests/utils/test_logging.py\n```\n\n### Sending test report to online pastebin service\n\nCreating a URL for each test failure:\n\n```bash\npytest --pastebin=failed tests/utils/test_logging.py\n```\n\nThis will submit test run information to a remote Paste service and provide a URL for each failure. You may select\ntests as usual or add for example -x if you only want to send one particular failure.\n\nCreating a URL for a whole test session log:\n\n```bash\npytest --pastebin=all tests/utils/test_logging.py\n```\n\n## Writing tests\n\n\ud83e\udd17 transformers tests are based on `unittest`, but run by `pytest`, so most of the time features from both systems\ncan be used.\n\nYou can read [here](https://docs.pytest.org/en/stable/unittest.html) which features are supported, but the important\nthing to remember is that most `pytest` fixtures don't work. Neither parametrization, but we use the module\n`parameterized` that works in a similar way.\n\n\n### Parametrization\n\nOften, there is a need to run the same test multiple times, but with different arguments. It could be done from within\nthe test, but then there is no way of running that test for just one set of arguments.\n\n```python\n# test_this1.py\nimport unittest\nfrom parameterized import parameterized\n\n\nclass TestMathUnitTest(unittest.TestCase):\n    @parameterized.expand(\n        [\n            (\"negative\", -1.5, -2.0),\n            (\"integer\", 1, 1.0),\n            (\"large fraction\", 1.6, 1),\n        ]\n    )\n    def test_floor(self, name, input, expected):\n        assert_equal(math.floor(input), expected)\n```\n\nNow, by default this test will be run 3 times, each time with the last 3 arguments of `test_floor` being assigned the\ncorresponding arguments in the parameter list.\n\nand you could run just the `negative` and `integer` sets of params with:\n\n```bash\npytest -k \"negative and integer\" tests/test_mytest.py\n```\n\nor all but `negative` sub-tests, with:\n\n```bash\npytest -k \"not negative\" tests/test_mytest.py\n```\n\nBesides using the `-k` filter that was just mentioned, you can find out the exact name of each sub-test and run any\nor all of them using their exact names.\n\n```bash\npytest test_this1.py --collect-only -q\n```\n\nand it will list:\n\n```bash\ntest_this1.py::TestMathUnitTest::test_floor_0_negative\ntest_this1.py::TestMathUnitTest::test_floor_1_integer\ntest_this1.py::TestMathUnitTest::test_floor_2_large_fraction\n```\n\nSo now you can run just 2 specific sub-tests:\n\n```bash\npytest test_this1.py::TestMathUnitTest::test_floor_0_negative  test_this1.py::TestMathUnitTest::test_floor_1_integer\n```\n\nThe module [parameterized](https://pypi.org/project/parameterized/) which is already in the developer dependencies\nof `transformers` works for both: `unittests` and `pytest` tests.\n\nIf, however, the test is not a `unittest`, you may use `pytest.mark.parametrize` (or you may see it being used in\nsome existing tests, mostly under `examples`).\n\nHere is the same example, this time using `pytest`'s `parametrize` marker:\n\n```python\n# test_this2.py\nimport pytest"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 74f59c67-d4f1-47e6-9e58-13fa8fa7b7ba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"74f59c67-d4f1-47e6-9e58-13fa8fa7b7ba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c1ff302-bbd4-4b30-a285-559358028aec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d7998983019ff1b1442b6664765c84ff27a0599d4fcf83ce80bd5c539556a469\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3ba4bcfb-a687-476f-9858-dd3fab37ff11\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e1a9f85e7d761ccd28d2f659c8ec5ad1997f84394b3c97b38fde0f1b736b7a44\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21663, \"end_char_idx\": 24324, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "@pytest.mark.parametrize(\n    \"name, input, expected\",\n    [\n        (\"negative\", -1.5, -2.0),\n        (\"integer\", 1, 1.0),\n        (\"large fraction\", 1.6, 1),\n    ],\n)\ndef test_floor(name, input, expected):\n    assert_equal(math.floor(input), expected)\n```\n\nSame as with `parameterized`, with `pytest.mark.parametrize` you can have a fine control over which sub-tests are\nrun, if the `-k` filter doesn't do the job. Except, this parametrization function creates a slightly different set of\nnames for the sub-tests. Here is what they look like:\n\n```bash\npytest test_this2.py --collect-only -q\n```\n\nand it will list:\n\n```bash\ntest_this2.py::test_floor[integer-1-1.0]\ntest_this2.py::test_floor[negative--1.5--2.0]\ntest_this2.py::test_floor[large fraction-1.6-1]\n```\n\nSo now you can run just the specific test:\n\n```bash\npytest test_this2.py::test_floor[negative--1.5--2.0] test_this2.py::test_floor[integer-1-1.0]\n```\n\nas in the previous example.\n\n\n\n### Files and directories\n\nIn tests often we need to know where things are relative to the current test file, and it's not trivial since the test\ncould be invoked from more than one directory or could reside in sub-directories with different depths. A helper class\n`transformers.test_utils.TestCasePlus` solves this problem by sorting out all the basic paths and provides easy\naccessors to them:\n\n- `pathlib` objects (all fully resolved):\n\n  - `test_file_path` - the current test file path, i.e. `__file__`\n  - `test_file_dir` - the directory containing the current test file\n  - `tests_dir` - the directory of the `tests` test suite\n  - `examples_dir` - the directory of the `examples` test suite\n  - `repo_root_dir` - the directory of the repository\n  - `src_dir` - the directory of `src` (i.e. where the `transformers` sub-dir resides)\n\n- stringified paths---same as above but these return paths as strings, rather than `pathlib` objects:\n\n  - `test_file_path_str`\n  - `test_file_dir_str`\n  - `tests_dir_str`\n  - `examples_dir_str`\n  - `repo_root_dir_str`\n  - `src_dir_str`\n\nTo start using those all you need is to make sure that the test resides in a subclass of\n`transformers.test_utils.TestCasePlus`. For example:\n\n```python\nfrom transformers.testing_utils import TestCasePlus\n\n\nclass PathExampleTest(TestCasePlus):\n    def test_something_involving_local_locations(self):\n        data_dir = self.tests_dir / \"fixtures/tests_samples/wmt_en_ro\"\n```\n\nIf you don't need to manipulate paths via `pathlib` or you just need a path as a string, you can always invoked\n`str()` on the `pathlib` object or use the accessors ending with `_str`. For example:\n\n```python\nfrom transformers.testing_utils import TestCasePlus"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3ba4bcfb-a687-476f-9858-dd3fab37ff11
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"3ba4bcfb-a687-476f-9858-dd3fab37ff11\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"74f59c67-d4f1-47e6-9e58-13fa8fa7b7ba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"6ff4dc18fca17dbfc1bf393ed894afe1d091141947284a2e462763e109a8ea72\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"037c103c-72ea-425c-94ac-de1e90d90183\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2fb931b43c9ad8db93408479f62947c456c4199844cd902abdc43be80ef1ecba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23895, \"end_char_idx\": 27221, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "class PathExampleTest(TestCasePlus):\n    def test_something_involving_local_locations(self):\n        data_dir = self.tests_dir / \"fixtures/tests_samples/wmt_en_ro\"\n```\n\nIf you don't need to manipulate paths via `pathlib` or you just need a path as a string, you can always invoked\n`str()` on the `pathlib` object or use the accessors ending with `_str`. For example:\n\n```python\nfrom transformers.testing_utils import TestCasePlus\n\n\nclass PathExampleTest(TestCasePlus):\n    def test_something_involving_stringified_locations(self):\n        examples_dir = self.examples_dir_str\n```\n\n### Temporary files and directories\n\nUsing unique temporary files and directories are essential for parallel test running, so that the tests won't overwrite\neach other's data. Also we want to get the temporary files and directories removed at the end of each test that created\nthem. Therefore, using packages like `tempfile`, which address these needs is essential.\n\nHowever, when debugging tests, you need to be able to see what goes into the temporary file or directory and you want\nto know it's exact path and not having it randomized on every test re-run.\n\nA helper class `transformers.test_utils.TestCasePlus` is best used for such purposes. It's a sub-class of\n`unittest.TestCase`, so we can easily inherit from it in the test modules.\n\nHere is an example of its usage:\n\n```python\nfrom transformers.testing_utils import TestCasePlus\n\n\nclass ExamplesTests(TestCasePlus):\n    def test_whatever(self):\n        tmp_dir = self.get_auto_remove_tmp_dir()\n```\n\nThis code creates a unique temporary directory, and sets `tmp_dir` to its location.\n\n- Create a unique temporary dir:\n\n```python\ndef test_whatever(self):\n    tmp_dir = self.get_auto_remove_tmp_dir()\n```\n\n`tmp_dir` will contain the path to the created temporary dir. It will be automatically removed at the end of the\ntest.\n\n- Create a temporary dir of my choice, ensure it's empty before the test starts and don't empty it after the test.\n\n```python\ndef test_whatever(self):\n    tmp_dir = self.get_auto_remove_tmp_dir(\"./xxx\")\n```\n\nThis is useful for debug when you want to monitor a specific directory and want to make sure the previous tests didn't\nleave any data in there.\n\n- You can override the default behavior by directly overriding the `before` and `after` args, leading to one of the\n  following behaviors:\n\n  - `before=True`: the temporary dir will always be cleared at the beginning of the test.\n  - `before=False`: if the temporary dir already existed, any existing files will remain there.\n  - `after=True`: the temporary dir will always be deleted at the end of the test.\n  - `after=False`: the temporary dir will always be left intact at the end of the test.\n\n<Tip>\n\nIn order to run the equivalent of `rm -r` safely, only subdirs of the project repository checkout are allowed if\nan explicit `tmp_dir` is used, so that by mistake no `/tmp` or similar important part of the filesystem will\nget nuked. i.e. please always pass paths that start with `./`.\n\n</Tip>\n\n<Tip>\n\nEach test can register multiple temporary directories and they all will get auto-removed, unless requested\notherwise.\n\n</Tip>\n\n### Temporary sys.path override\n\nIf you need to temporary override `sys.path` to import from another test for example, you can use the\n`ExtendSysPath` context manager. Example:"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 037c103c-72ea-425c-94ac-de1e90d90183
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"037c103c-72ea-425c-94ac-de1e90d90183\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3ba4bcfb-a687-476f-9858-dd3fab37ff11\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"9f59e15aa688cfbaa7d65a69757d4be2951052c84ea82fbe620dcb7e1c98c728\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f6986bba-5325-45b4-9050-33e1a53c325e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"74d4c3e2ea345f79bf0dbdf2c36584b8b9798f3ded4c59ca4c7ecd01471c3a58\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27224, \"end_char_idx\": 30254, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "```python\nimport os\nfrom transformers.testing_utils import ExtendSysPath\n\nbindir = os.path.abspath(os.path.dirname(__file__))\nwith ExtendSysPath(f\"{bindir}/..\"):\n    from test_trainer import TrainerIntegrationCommon  # noqa\n```\n\n### Skipping tests\n\nThis is useful when a bug is found and a new test is written, yet the bug is not fixed yet. In order to be able to\ncommit it to the main repository we need make sure it's skipped during `make test`.\n\nMethods:\n\n-  A **skip** means that you expect your test to pass only if some conditions are met, otherwise pytest should skip\n  running the test altogether. Common examples are skipping windows-only tests on non-windows platforms, or skipping\n  tests that depend on an external resource which is not available at the moment (for example a database).\n\n-  A **xfail** means that you expect a test to fail for some reason. A common example is a test for a feature not yet\n  implemented, or a bug not yet fixed. When a test passes despite being expected to fail (marked with\n  pytest.mark.xfail), it\u2019s an xpass and will be reported in the test summary.\n\nOne of the important differences between the two is that `skip` doesn't run the test, and `xfail` does. So if the\ncode that's buggy causes some bad state that will affect other tests, do not use `xfail`.\n\n#### Implementation\n\n- Here is how to skip whole test unconditionally:\n\n```python no-style\n@unittest.skip(reason=\"this bug needs to be fixed\")\ndef test_feature_x():\n```\n\nor via pytest:\n\n```python no-style\n@pytest.mark.skip(reason=\"this bug needs to be fixed\")\n```\n\nor the `xfail` way:\n\n```python no-style\n@pytest.mark.xfail\ndef test_feature_x():\n```\n\n\nHere's how to skip a test based on internal checks within the test:\n\n```python\ndef test_feature_x():\n    if not has_something():\n        pytest.skip(\"unsupported configuration\")\n```\n\nor the whole module:\n\n```python\nimport pytest\n\nif not pytest.config.getoption(\"--custom-flag\"):\n    pytest.skip(\"--custom-flag is missing, skipping tests\", allow_module_level=True)\n```\n\nor the `xfail` way:\n\n```python\ndef test_feature_x():\n    pytest.xfail(\"expected to fail until bug XYZ is fixed\")\n```\n\n- Here is how to skip all tests in a module if some import is missing:\n\n```python\ndocutils = pytest.importorskip(\"docutils\", minversion=\"0.3\")\n```\n\n-  Skip a test based on a condition:\n\n```python no-style\n@pytest.mark.skipif(sys.version_info < (3,6), reason=\"requires python3.6 or higher\")\ndef test_feature_x():\n```\n\nor:\n\n```python no-style\n@unittest.skipIf(torch_device == \"cpu\", \"Can't do half precision\")\ndef test_feature_x():\n```\n\nor skip the whole module:\n\n```python no-style\n@pytest.mark.skipif(sys.platform == 'win32', reason=\"does not run on windows\")\nclass TestClass():\n    def test_feature_x(self):\n```\n\nMore details, example and ways are [here](https://docs.pytest.org/en/latest/skipping.html).\n\n### Slow tests\n\nThe library of tests is ever-growing, and some of the tests take minutes to run, therefore we can't afford waiting for\nan hour for the test suite to complete on CI."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f6986bba-5325-45b4-9050-33e1a53c325e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"f6986bba-5325-45b4-9050-33e1a53c325e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"037c103c-72ea-425c-94ac-de1e90d90183\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"af9dbd5c62ff9d09d624cf59077f5800ab1b6631d8b52817a3afa288c1a17ac9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"edfba35c-0411-4347-bbdd-6fb6d9fe410c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"68842981d9d4c92acafa2028312c1a402e6a9f2898f5b65b69f4b1caaa5936d2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 28880, \"end_char_idx\": 32132, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "Here's how to skip a test based on internal checks within the test:\n\n```python\ndef test_feature_x():\n    if not has_something():\n        pytest.skip(\"unsupported configuration\")\n```\n\nor the whole module:\n\n```python\nimport pytest\n\nif not pytest.config.getoption(\"--custom-flag\"):\n    pytest.skip(\"--custom-flag is missing, skipping tests\", allow_module_level=True)\n```\n\nor the `xfail` way:\n\n```python\ndef test_feature_x():\n    pytest.xfail(\"expected to fail until bug XYZ is fixed\")\n```\n\n- Here is how to skip all tests in a module if some import is missing:\n\n```python\ndocutils = pytest.importorskip(\"docutils\", minversion=\"0.3\")\n```\n\n-  Skip a test based on a condition:\n\n```python no-style\n@pytest.mark.skipif(sys.version_info < (3,6), reason=\"requires python3.6 or higher\")\ndef test_feature_x():\n```\n\nor:\n\n```python no-style\n@unittest.skipIf(torch_device == \"cpu\", \"Can't do half precision\")\ndef test_feature_x():\n```\n\nor skip the whole module:\n\n```python no-style\n@pytest.mark.skipif(sys.platform == 'win32', reason=\"does not run on windows\")\nclass TestClass():\n    def test_feature_x(self):\n```\n\nMore details, example and ways are [here](https://docs.pytest.org/en/latest/skipping.html).\n\n### Slow tests\n\nThe library of tests is ever-growing, and some of the tests take minutes to run, therefore we can't afford waiting for\nan hour for the test suite to complete on CI. Therefore, with some exceptions for essential tests, slow tests should be\nmarked as in the example below:\n\n```python no-style\nfrom transformers.testing_utils import slow\n@slow\ndef test_integration_foo():\n```\n\nOnce a test is marked as `@slow`, to run such tests set `RUN_SLOW=1` env var, e.g.:\n\n```bash\nRUN_SLOW=1 pytest tests\n```\n\nSome decorators like `@parameterized` rewrite test names, therefore `@slow` and the rest of the skip decorators\n`@require_*` have to be listed last for them to work correctly. Here is an example of the correct usage:\n\n```python no-style\n@parameterized.expand(...)\n@slow\ndef test_integration_foo():\n```\n\nAs explained at the beginning of this document, slow tests get to run on a scheduled basis, rather than in PRs CI\nchecks. So it's possible that some problems will be missed during a PR submission and get merged. Such problems will\nget caught during the next scheduled CI job. But it also means that it's important to run the slow tests on your\nmachine before submitting the PR.\n\nHere is a rough decision making mechanism for choosing which tests should be marked as slow:\n\nIf the test is focused on one of the library's internal components (e.g., modeling files, tokenization files,\npipelines), then we should run that test in the non-slow test suite. If it's focused on an other aspect of the library,\nsuch as the documentation or the examples, then we should run these tests in the slow test suite. And then, to refine\nthis approach we should have exceptions:\n\n- All tests that need to download a heavy set of weights or a dataset that is larger than ~50MB (e.g., model or\n  tokenizer integration tests, pipeline integration tests) should be set to slow. If you're adding a new model, you\n  should create and upload to the hub a tiny version of it (with random weights) for integration tests. This is\n  discussed in the following paragraphs."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: edfba35c-0411-4347-bbdd-6fb6d9fe410c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"edfba35c-0411-4347-bbdd-6fb6d9fe410c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6986bba-5325-45b4-9050-33e1a53c325e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"790e07a5719e85d347f4f4bc3f7299851aa548eb9e30fb5340c9bd05623ddaa5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9e1bf86c-d54b-4557-8231-b49af58466cb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0007f7e2b27124ebd7f88da8ae267c5696e33e0ac98446b4dfa78b0544a036e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30546, \"end_char_idx\": 33817, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": ":\n\n```bash\nRUN_SLOW=1 pytest tests\n```\n\nSome decorators like `@parameterized` rewrite test names, therefore `@slow` and the rest of the skip decorators\n`@require_*` have to be listed last for them to work correctly. Here is an example of the correct usage:\n\n```python no-style\n@parameterized.expand(...)\n@slow\ndef test_integration_foo():\n```\n\nAs explained at the beginning of this document, slow tests get to run on a scheduled basis, rather than in PRs CI\nchecks. So it's possible that some problems will be missed during a PR submission and get merged. Such problems will\nget caught during the next scheduled CI job. But it also means that it's important to run the slow tests on your\nmachine before submitting the PR.\n\nHere is a rough decision making mechanism for choosing which tests should be marked as slow:\n\nIf the test is focused on one of the library's internal components (e.g., modeling files, tokenization files,\npipelines), then we should run that test in the non-slow test suite. If it's focused on an other aspect of the library,\nsuch as the documentation or the examples, then we should run these tests in the slow test suite. And then, to refine\nthis approach we should have exceptions:\n\n- All tests that need to download a heavy set of weights or a dataset that is larger than ~50MB (e.g., model or\n  tokenizer integration tests, pipeline integration tests) should be set to slow. If you're adding a new model, you\n  should create and upload to the hub a tiny version of it (with random weights) for integration tests. This is\n  discussed in the following paragraphs.\n- All tests that need to do a training not specifically optimized to be fast should be set to slow.\n- We can introduce exceptions if some of these should-be-non-slow tests are excruciatingly slow, and set them to\n  `@slow`. Auto-modeling tests, which save and load large files to disk, are a good example of tests that are marked\n  as `@slow`.\n- If a test completes under 1 second on CI (including downloads if any) then it should be a normal test regardless.\n\nCollectively, all the non-slow tests need to cover entirely the different internals, while remaining fast. For example,\na significant coverage can be achieved by testing with specially created tiny models with random weights. Such models\nhave the very minimal number of layers (e.g., 2), vocab size (e.g., 1000), etc. Then the `@slow` tests can use large\nslow models to do qualitative testing. To see the use of these simply look for *tiny* models with:\n\n```bash\ngrep tiny tests examples\n```\n\nHere is an example of a [script](https://github.com/huggingface/transformers/tree/main/scripts/fsmt/fsmt-make-tiny-model.py) that created the tiny model\n[stas/tiny-wmt19-en-de](https://huggingface.co/stas/tiny-wmt19-en-de). You can easily adjust it to your specific\nmodel's architecture.\n\nIt's easy to measure the run-time incorrectly if for example there is an overheard of downloading a huge model, but if\nyou test it locally the downloaded files would be cached and thus the download time not measured. Hence check the\nexecution speed report in CI logs instead (the output of `pytest --durations=0 tests`).\n\nThat report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9e1bf86c-d54b-4557-8231-b49af58466cb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"9e1bf86c-d54b-4557-8231-b49af58466cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"edfba35c-0411-4347-bbdd-6fb6d9fe410c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"7d777b33761de1223c09f1babb7f1816a957ef7c70c878d8034c52392f7bc300\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"331cfaba-085c-4701-a699-83e799c98c2c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bb8fd2419b90aa9446a4b7ceda7da957800c8e340f9d0aa618d7dc27f5e51d88\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32233, \"end_char_idx\": 35277, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "- We can introduce exceptions if some of these should-be-non-slow tests are excruciatingly slow, and set them to\n  `@slow`. Auto-modeling tests, which save and load large files to disk, are a good example of tests that are marked\n  as `@slow`.\n- If a test completes under 1 second on CI (including downloads if any) then it should be a normal test regardless.\n\nCollectively, all the non-slow tests need to cover entirely the different internals, while remaining fast. For example,\na significant coverage can be achieved by testing with specially created tiny models with random weights. Such models\nhave the very minimal number of layers (e.g., 2), vocab size (e.g., 1000), etc. Then the `@slow` tests can use large\nslow models to do qualitative testing. To see the use of these simply look for *tiny* models with:\n\n```bash\ngrep tiny tests examples\n```\n\nHere is an example of a [script](https://github.com/huggingface/transformers/tree/main/scripts/fsmt/fsmt-make-tiny-model.py) that created the tiny model\n[stas/tiny-wmt19-en-de](https://huggingface.co/stas/tiny-wmt19-en-de). You can easily adjust it to your specific\nmodel's architecture.\n\nIt's easy to measure the run-time incorrectly if for example there is an overheard of downloading a huge model, but if\nyou test it locally the downloaded files would be cached and thus the download time not measured. Hence check the\nexecution speed report in CI logs instead (the output of `pytest --durations=0 tests`).\n\nThat report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast.\nIf you notice that the test suite starts getting slow on CI, the top listing of this report will show the slowest\ntests.\n\n\n### Testing the stdout/stderr output\n\nIn order to test functions that write to `stdout` and/or `stderr`, the test can access those streams using the\n`pytest`'s [capsys system](https://docs.pytest.org/en/latest/capture.html). Here is how this is accomplished:\n\n```python\nimport sys\n\n\ndef print_to_stdout(s):\n    print(s)\n\n\ndef print_to_stderr(s):\n    sys.stderr.write(s)\n\n\ndef test_result_and_stdout(capsys):\n    msg = \"Hello\"\n    print_to_stdout(msg)\n    print_to_stderr(msg)\n    out, err = capsys.readouterr()  # consume the captured output streams\n    # optional: if you want to replay the consumed streams:\n    sys.stdout.write(out)\n    sys.stderr.write(err)\n    # test:\n    assert msg in out\n    assert msg in err\n```\n\nAnd, of course, most of the time, `stderr` will come as a part of an exception, so try/except has to be used in such\na case:\n\n```python\ndef raise_exception(msg):\n    raise ValueError(msg)\n\n\ndef test_something_exception():\n    msg = \"Not a good value\"\n    error = \"\"\n    try:\n        raise_exception(msg)\n    except Exception as e:\n        error = str(e)\n        assert msg in error, f\"{msg} is in the exception:\\n{error}\"\n```\n\nAnother approach to capturing stdout is via `contextlib.redirect_stdout`:\n\n```python\nfrom io import StringIO\nfrom contextlib import redirect_stdout\n\n\ndef print_to_stdout(s):\n    print(s)"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 331cfaba-085c-4701-a699-83e799c98c2c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"331cfaba-085c-4701-a699-83e799c98c2c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9e1bf86c-d54b-4557-8231-b49af58466cb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"57c52f13e7495f7081f9ce49f9189c31a292efa441236d5f8be97d9090a2d789\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"48adee9a-5814-4bdc-a1ba-af4a3606df36\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2edc364b2b73fe64690d6423c8613550bd19d2662997787614084d0db44a5db2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 33698, \"end_char_idx\": 37171, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "That report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast.\nIf you notice that the test suite starts getting slow on CI, the top listing of this report will show the slowest\ntests.\n\n\n### Testing the stdout/stderr output\n\nIn order to test functions that write to `stdout` and/or `stderr`, the test can access those streams using the\n`pytest`'s [capsys system](https://docs.pytest.org/en/latest/capture.html). Here is how this is accomplished:\n\n```python\nimport sys\n\n\ndef print_to_stdout(s):\n    print(s)\n\n\ndef print_to_stderr(s):\n    sys.stderr.write(s)\n\n\ndef test_result_and_stdout(capsys):\n    msg = \"Hello\"\n    print_to_stdout(msg)\n    print_to_stderr(msg)\n    out, err = capsys.readouterr()  # consume the captured output streams\n    # optional: if you want to replay the consumed streams:\n    sys.stdout.write(out)\n    sys.stderr.write(err)\n    # test:\n    assert msg in out\n    assert msg in err\n```\n\nAnd, of course, most of the time, `stderr` will come as a part of an exception, so try/except has to be used in such\na case:\n\n```python\ndef raise_exception(msg):\n    raise ValueError(msg)\n\n\ndef test_something_exception():\n    msg = \"Not a good value\"\n    error = \"\"\n    try:\n        raise_exception(msg)\n    except Exception as e:\n        error = str(e)\n        assert msg in error, f\"{msg} is in the exception:\\n{error}\"\n```\n\nAnother approach to capturing stdout is via `contextlib.redirect_stdout`:\n\n```python\nfrom io import StringIO\nfrom contextlib import redirect_stdout\n\n\ndef print_to_stdout(s):\n    print(s)\n\n\ndef test_result_and_stdout():\n    msg = \"Hello\"\n    buffer = StringIO()\n    with redirect_stdout(buffer):\n        print_to_stdout(msg)\n    out = buffer.getvalue()\n    # optional: if you want to replay the consumed streams:\n    sys.stdout.write(out)\n    # test:\n    assert msg in out\n```\n\nAn important potential issue with capturing stdout is that it may contain `\\r` characters that in normal `print`\nreset everything that has been printed so far. There is no problem with `pytest`, but with `pytest -s` these\ncharacters get included in the buffer, so to be able to have the test run with and without `-s`, you have to make an\nextra cleanup to the captured output, using `re.sub(r'~.*\\r', '', buf, 0, re.M)`.\n\nBut, then we have a helper context manager wrapper to automatically take care of it all, regardless of whether it has\nsome `\\r`'s in it or not, so it's a simple:\n\n```python\nfrom transformers.testing_utils import CaptureStdout\n\nwith CaptureStdout() as cs:\n    function_that_writes_to_stdout()\nprint(cs.out)\n```\n\nHere is a full test example:\n\n```python\nfrom transformers.testing_utils import CaptureStdout\n\nmsg = \"Secret message\\r\"\nfinal = \"Hello World\"\nwith CaptureStdout() as cs:\n    print(msg + final)\nassert cs.out == final + \"\\n\", f\"captured: {cs.out}, expecting {final}\"\n```\n\nIf you'd like to capture `stderr` use the `CaptureStderr` class instead:\n\n```python\nfrom transformers.testing_utils import CaptureStderr\n\nwith CaptureStderr() as cs:\n    function_that_writes_to_stderr()\nprint(cs.err)\n```\n\nIf you need to capture both streams at once, use the parent `CaptureStd` class:\n\n```python\nfrom transformers.testing_utils import CaptureStd\n\nwith CaptureStd() as cs:\n    function_that_writes_to_stdout_and_stderr()\nprint(cs.err, cs.out)\n```\n\nAlso, to aid debugging test issues, by default these context managers automatically replay the captured streams on exit\nfrom the context."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 48adee9a-5814-4bdc-a1ba-af4a3606df36
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"48adee9a-5814-4bdc-a1ba-af4a3606df36\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"331cfaba-085c-4701-a699-83e799c98c2c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"a59549a3ca5c0f17b1860811e6b39dbb20faeae3eef13834014c7ff974fd223c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"71bcfb87-5676-45bd-ae10-accc101a3b24\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f45734368c77a8ef8f506540e9f9f591146d3397d76095912bb02705661f4375\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 37174, \"end_char_idx\": 40305, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "### Capturing logger stream\n\nIf you need to validate the output of a logger, you can use `CaptureLogger`:\n\n```python\nfrom transformers import logging\nfrom transformers.testing_utils import CaptureLogger\n\nmsg = \"Testing 1, 2, 3\"\nlogging.set_verbosity_info()\nlogger = logging.get_logger(\"transformers.models.bart.tokenization_bart\")\nwith CaptureLogger(logger) as cl:\n    logger.info(msg)\nassert cl.out, msg + \"\\n\"\n```\n\n### Testing with environment variables\n\nIf you want to test the impact of environment variables for a specific test you can use a helper decorator\n`transformers.testing_utils.mockenv`\n\n```python\nfrom transformers.testing_utils import mockenv\n\n\nclass HfArgumentParserTest(unittest.TestCase):\n    @mockenv(TRANSFORMERS_VERBOSITY=\"error\")\n    def test_env_override(self):\n        env_level_str = os.getenv(\"TRANSFORMERS_VERBOSITY\", None)\n```\n\nAt times an external program needs to be called, which requires setting `PYTHONPATH` in `os.environ` to include\nmultiple local paths. A helper class `transformers.test_utils.TestCasePlus` comes to help:\n\n```python\nfrom transformers.testing_utils import TestCasePlus\n\n\nclass EnvExampleTest(TestCasePlus):\n    def test_external_prog(self):\n        env = self.get_env()\n        # now call the external program, passing `env` to it\n```\n\nDepending on whether the test file was under the `tests` test suite or `examples` it'll correctly set up\n`env[PYTHONPATH]` to include one of these two directories, and also the `src` directory to ensure the testing is\ndone against the current repo, and finally with whatever `env[PYTHONPATH]` was already set to before the test was\ncalled if anything.\n\nThis helper method creates a copy of the `os.environ` object, so the original remains intact.\n\n\n### Getting reproducible results\n\nIn some situations you may want to remove randomness for your tests. To get identical reproducible results set, you\nwill need to fix the seed:\n\n```python\nseed = 42\n\n# python RNG\nimport random\n\nrandom.seed(seed)\n\n# pytorch RNGs\nimport torch\n\ntorch.manual_seed(seed)\ntorch.backends.cudnn.deterministic = True\nif torch.cuda.is_available():\n    torch.cuda.manual_seed_all(seed)\n\n# numpy RNG\nimport numpy as np\n\nnp.random.seed(seed)\n\n# tf RNG\ntf.random.set_seed(seed)\n```\n\n### Debugging tests\n\nTo start a debugger at the point of the warning, do this:\n\n```bash\npytest tests/utils/test_logging.py -W error::UserWarning --pdb\n```\n\n## Working with github actions workflows\n\nTo trigger a self-push workflow CI job, you must:\n\n1. Create a new branch on `transformers` origin (not a fork!).\n2. The branch name has to start with either `ci_` or `ci-` (`main` triggers it too, but we can't do PRs on\n   `main`). It also gets triggered only for specific paths - you can find the up-to-date definition in case it\n   changed since this document has been written [here](https://github.com/huggingface/transformers/blob/main/.github/workflows/self-push.yml) under *push:*\n3. Create a PR from this branch.\n4. Then you can see the job appear [here](https://github.com/huggingface/transformers/actions/workflows/self-push.yml). It may not run right away if there\n   is a backlog."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 71bcfb87-5676-45bd-ae10-accc101a3b24
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"71bcfb87-5676-45bd-ae10-accc101a3b24\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0c911e2-c921-49f6-af6a-9a76615097a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dc311cb42ecc98b80381bb1a01c9655909cc2e78436aa3454c2fa7e1d74c9000\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"48adee9a-5814-4bdc-a1ba-af4a3606df36\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/testing\", \"title\": \"Testing\", \"tokens\": 10558, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"77da86d73070f44c6e607bf2890cd9ff29725400656a0462294a1a6946bdc18f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 38913, \"end_char_idx\": 43501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "ref_doc_id": "f0c911e2-c921-49f6-af6a-9a76615097a7", "chroma:document": "### Getting reproducible results\n\nIn some situations you may want to remove randomness for your tests. To get identical reproducible results set, you\nwill need to fix the seed:\n\n```python\nseed = 42\n\n# python RNG\nimport random\n\nrandom.seed(seed)\n\n# pytorch RNGs\nimport torch\n\ntorch.manual_seed(seed)\ntorch.backends.cudnn.deterministic = True\nif torch.cuda.is_available():\n    torch.cuda.manual_seed_all(seed)\n\n# numpy RNG\nimport numpy as np\n\nnp.random.seed(seed)\n\n# tf RNG\ntf.random.set_seed(seed)\n```\n\n### Debugging tests\n\nTo start a debugger at the point of the warning, do this:\n\n```bash\npytest tests/utils/test_logging.py -W error::UserWarning --pdb\n```\n\n## Working with github actions workflows\n\nTo trigger a self-push workflow CI job, you must:\n\n1. Create a new branch on `transformers` origin (not a fork!).\n2. The branch name has to start with either `ci_` or `ci-` (`main` triggers it too, but we can't do PRs on\n   `main`). It also gets triggered only for specific paths - you can find the up-to-date definition in case it\n   changed since this document has been written [here](https://github.com/huggingface/transformers/blob/main/.github/workflows/self-push.yml) under *push:*\n3. Create a PR from this branch.\n4. Then you can see the job appear [here](https://github.com/huggingface/transformers/actions/workflows/self-push.yml). It may not run right away if there\n   is a backlog.\n\n\n\n\n## Testing Experimental CI Features\n\nTesting CI features can be potentially problematic as it can interfere with the normal CI functioning. Therefore if a\nnew CI feature is to be added, it should be done as following.\n\n1. Create a new dedicated job that tests what needs to be tested\n2. The new job must always succeed so that it gives us a green \u2713 (details below).\n3. Let it run for some days to see that a variety of different PR types get to run on it (user fork branches,\n   non-forked branches, branches originating from github.com UI direct file edit, various forced pushes, etc. - there\n   are so many) while monitoring the experimental job's logs (not the overall job green as it's purposefully always\n   green)\n4. When it's clear that everything is solid, then merge the new changes into existing jobs.\n\nThat way experiments on CI functionality itself won't interfere with the normal workflow.\n\nNow how can we make the job always succeed while the new CI feature is being developed?\n\nSome CIs, like TravisCI support ignore-step-failure and will report the overall job as successful, but CircleCI and\nGithub Actions as of this writing don't support that.\n\nSo the following workaround can be used:\n\n1. `set +euo pipefail` at the beginning of the run command to suppress most potential failures in the bash script.\n2. the last command must be a success: `echo \"done\"` or just `true` will do\n\nHere is an example:\n\n```yaml\n- run:\n    name: run CI experiment\n    command: |\n        set +euo pipefail\n        echo \"setting run-all-despite-any-errors-mode\"\n        this_command_will_fail\n        echo \"but bash continues to run\"\n        # emulate another failure\n        false\n        # but the last command must be a success\n        echo \"during experiment do not remove: reporting success to CI, even if there were failures\"\n```\n\nFor simple commands you could also do:\n\n```bash\ncmd_that_may_fail || true\n```\n\nOf course, once satisfied with the results, integrate the experimental step or job with the rest of the normal jobs,\nwhile removing `set +euo pipefail` or any other things you may have added to ensure that the experimental job doesn't\ninterfere with the normal CI functioning.\n\nThis whole process would have been much easier if we only could set something like `allow-failure` for the\nexperimental step, and let it fail without impacting the overall status of PRs. But as mentioned earlier CircleCI and\nGithub Actions don't support it at the moment.\n\nYou can vote for this feature and see where it is at these CI-specific threads:\n\n- [Github Actions:](https://github.com/actions/toolkit/issues/399)\n- [CircleCI:](https://ideas.circleci.com/ideas/CCI-I-344)\n\n## DeepSpeed integration\n\nFor a PR that involves the DeepSpeed integration, keep in mind our CircleCI PR CI setup doesn't have GPUs. Tests requiring GPUs are run on a different CI nightly. This means if you get a passing CI report in your PR, it doesn\u2019t mean the DeepSpeed tests pass.\n\nTo run DeepSpeed tests:\n\n```bash\nRUN_SLOW=1 pytest tests/deepspeed/test_deepspeed.py\n```\n\nAny changes to the modeling or PyTorch examples code requires running the model zoo tests as well.\n\n```bash\nRUN_SLOW=1 pytest tests/deepspeed\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2691a3e4-32a3-4cd7-8367-9b6ba3fa409e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2691a3e4-32a3-4cd7-8367-9b6ba3fa409e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/performance\", \"title\": \"Performance and Scalability\", \"tokens\": 838, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9222a9f9-7666-49c0-ab16-8394389abd03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/performance\", \"title\": \"Performance and Scalability\", \"tokens\": 838, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4080bf02bd42c08da311b192ac1c7aa133ecbf6a8a68562c9f70249f67176977\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ef4a466c-9ffa-4d90-bd1f-308dabd121ac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2630005ea254b3a44b9e379bdcc05906a643079260b0894c7643a80596470c4a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3352, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9222a9f9-7666-49c0-ab16-8394389abd03", "doc_id": "9222a9f9-7666-49c0-ab16-8394389abd03", "ref_doc_id": "9222a9f9-7666-49c0-ab16-8394389abd03", "chroma:document": "<!---\nCopyright 2021 The HuggingFace Team. All rights reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n\u26a0\ufe0f Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be\nrendered properly in your Markdown viewer.\n\n-->\n\n# Performance and Scalability\n\nTraining large transformer models and deploying them to production present various challenges.  \nDuring training, the model may require more GPU memory than available or exhibit slow training speed. In the deployment \nphase, the model can struggle to handle the required throughput in a production environment.\n\nThis documentation aims to assist you in overcoming these challenges and finding the optimal setting for your use-case. \nThe guides are divided into training and inference sections, as each comes with different challenges and solutions. \nWithin each section you'll find separate guides for different hardware configurations, such as single GPU vs. multi-GPU \nfor training or CPU vs. GPU for inference.\n\nUse this document as your starting point to navigate further to the methods that match your scenario.\n\n## Training\n\nTraining large transformer models efficiently requires an accelerator such as a GPU or TPU. The most common case is where \nyou have a single GPU. The methods that you can apply to improve training efficiency on a single GPU extend to other setups \nsuch as multiple GPU. However, there are also techniques that are specific to multi-GPU or CPU training. We cover them in \nseparate sections.\n\n* [Methods and tools for efficient training on a single GPU](perf_train_gpu_one): start here to learn common approaches that can help optimize GPU memory utilization, speed up the training, or both. \n* [Multi-GPU training section](perf_train_gpu_many): explore this section to learn about further optimization methods that apply to a multi-GPU settings, such as data, tensor, and pipeline parallelism.\n* [CPU training section](perf_train_cpu): learn about mixed precision training on CPU.\n* [Efficient Training on Multiple CPUs](perf_train_cpu_many): learn about distributed CPU training.\n* [Training on TPU with TensorFlow](perf_train_tpu_tf): if you are new to TPUs, refer to this section for an opinionated introduction to training on TPUs and using XLA. \n* [Custom hardware for training](perf_hardware): find tips and tricks when building your own deep learning rig.\n* [Hyperparameter Search using Trainer API](hpo_train)\n\n## Inference\n\nEfficient inference with large models in a production environment can be as challenging as training them. In the following \nsections we go through the steps to run inference on CPU and single/multi-GPU setups.\n\n* [Inference on a single CPU](perf_infer_cpu)\n* [Inference on a single GPU](perf_infer_gpu_one)\n* [Multi-GPU inference](perf_infer_gpu_one)\n* [XLA Integration for TensorFlow Models](tf_xla)"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ef4a466c-9ffa-4d90-bd1f-308dabd121ac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ef4a466c-9ffa-4d90-bd1f-308dabd121ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/performance\", \"title\": \"Performance and Scalability\", \"tokens\": 838, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9222a9f9-7666-49c0-ab16-8394389abd03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/performance\", \"title\": \"Performance and Scalability\", \"tokens\": 838, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4080bf02bd42c08da311b192ac1c7aa133ecbf6a8a68562c9f70249f67176977\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2691a3e4-32a3-4cd7-8367-9b6ba3fa409e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/performance\", \"title\": \"Performance and Scalability\", \"tokens\": 838, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a0b9f20f17aacdbe8c4880837040b0b356989bde3873ad7a6d6fd883483d5b79\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3355, \"end_char_idx\": 4029, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9222a9f9-7666-49c0-ab16-8394389abd03", "doc_id": "9222a9f9-7666-49c0-ab16-8394389abd03", "ref_doc_id": "9222a9f9-7666-49c0-ab16-8394389abd03", "chroma:document": "## Training and inference\n\nHere you'll find techniques, tips and tricks that apply whether you are training a model, or running inference with it.\n\n* [Instantiating a big model](big_models)\n* [Troubleshooting performance issues](debugging)\n\n## Contribute\n\nThis document is far from being complete and a lot more needs to be added, so if you have additions or corrections to \nmake please don't hesitate to open a PR or if you aren't sure start an Issue and we can discuss the details there.\n\nWhen making contributions that A is better than B, please try to include a reproducible benchmark and/or a link to the \nsource of that information (unless it comes directly from you)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d01c7baa-83d9-41c6-938c-7263a2eebe66
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d01c7baa-83d9-41c6-938c-7263a2eebe66\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ff5a904-2dd0-42c3-ab52-3cc4a5c03cf7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"86b9274f60676ae660c2b6f1e47b78707b5ff3719e8f22033304d757e3dc5e0b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3580, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "# What \ud83e\udd17 Transformers can do\n\n\ud83e\udd17 Transformers is a library of pretrained state-of-the-art models for natural language processing (NLP), computer vision, and audio and speech processing tasks. Not only does the library contain Transformer models, but it also has non-Transformer models like modern convolutional networks for computer vision tasks. If you look at some of the most popular consumer products today, like smartphones, apps, and televisions, odds are that some kind of deep learning technology is behind it. Want to remove a background object from a picture taken by your smartphone? This is an example of a panoptic segmentation task (don't worry if you don't know what this means yet, we'll describe it in the following sections!). \n\nThis page provides an overview of the different speech and audio, computer vision, and NLP tasks that can be solved with the \ud83e\udd17 Transformers library in just three lines of code!\n\n## Audio\n\nAudio and speech processing tasks are a little different from the other modalities mainly because audio as an input is a continuous signal. Unlike text, a raw audio waveform can't be neatly split into discrete chunks the way a sentence can be divided into words. To get around this, the raw audio signal is typically sampled at regular intervals. If you take more samples within an interval, the sampling rate is higher, and the audio more closely resembles the original audio source.\n\nPrevious approaches preprocessed the audio to extract useful features from it. It is now more common to start audio and speech processing tasks by directly feeding the raw audio waveform to a feature encoder to extract an audio representation. This simplifies the preprocessing step and allows the model to learn the most essential features.\n\n### Audio classification\n\nAudio classification is a task that labels audio data from a predefined set of classes. It is a broad category with many specific applications, some of which include:\n\n* acoustic scene classification: label audio with a scene label (\"office\", \"beach\", \"stadium\")\n* acoustic event detection: label audio with a sound event label (\"car horn\", \"whale calling\", \"glass breaking\")\n* tagging: label audio containing multiple sounds (birdsongs, speaker identification in a meeting)\n* music classification: label music with a genre label (\"metal\", \"hip-hop\", \"country\")\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(task=\"audio-classification\", model=\"superb/hubert-base-superb-er\")\n>>> preds = classifier(\"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac\")\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"]} for pred in preds]\n>>> preds\n[{'score': 0.4532, 'label': 'hap'},\n {'score': 0.3622, 'label': 'sad'},\n {'score': 0.0943, 'label': 'neu'},\n {'score': 0.0903, 'label': 'ang'}]\n```\n\n### Automatic speech recognition\n\nAutomatic speech recognition (ASR) transcribes speech into text. It is one of the most common audio tasks due partly to speech being such a natural form of human communication. Today, ASR systems are embedded in \"smart\" technology products like speakers, phones, and cars. We can ask our virtual assistants to play music, set reminders, and tell us the weather. \n\nBut one of the key challenges Transformer architectures have helped with is in low-resource languages. By pretraining on large amounts of speech data, finetuning the model on only one hour of labeled speech data in a low-resource language can still produce high-quality results compared to previous ASR systems trained on 100x more labeled data."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4ff5a904-2dd0-42c3-ab52-3cc4a5c03cf7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4ff5a904-2dd0-42c3-ab52-3cc4a5c03cf7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d01c7baa-83d9-41c6-938c-7263a2eebe66\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"25eb3f029331abd1f689c5175e8884a2c8c5e1defc3ef469088cccf2fd86076a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c864edd4-68f8-4d85-a3ba-775c8aacc867\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4217f64a15a9898c2b855905d6c1a845687f16da3f284c4416b095abb8789ae4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2936, \"end_char_idx\": 6277, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "It is one of the most common audio tasks due partly to speech being such a natural form of human communication. Today, ASR systems are embedded in \"smart\" technology products like speakers, phones, and cars. We can ask our virtual assistants to play music, set reminders, and tell us the weather. \n\nBut one of the key challenges Transformer architectures have helped with is in low-resource languages. By pretraining on large amounts of speech data, finetuning the model on only one hour of labeled speech data in a low-resource language can still produce high-quality results compared to previous ASR systems trained on 100x more labeled data.\n\n```py\n>>> from transformers import pipeline\n\n>>> transcriber = pipeline(task=\"automatic-speech-recognition\", model=\"openai/whisper-small\")\n>>> transcriber(\"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac\")\n{'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'}\n```\n\n## Computer vision\n\nOne of the first and earliest successful computer vision tasks was recognizing images of zip code numbers using a [convolutional neural network (CNN)](glossary#convolution). An image is composed of pixels, and each pixel has a numerical value. This makes it easy to represent an image as a matrix of pixel values. Each particular combination of pixel values describes the colors of an image. \n\nTwo general ways computer vision tasks can be solved are:\n\n1. Use convolutions to learn the hierarchical features of an image from low-level features to high-level abstract things.\n2. Split an image into patches and use a Transformer to gradually learn how each image patch is related to each other to form an image. Unlike the bottom-up approach favored by a CNN, this is kind of like starting out with a blurry image and then gradually bringing it into focus.\n\n### Image classification\n\nImage classification labels an entire image from a predefined set of classes. Like most classification tasks, there are many practical use cases for image classification, some of which include:\n\n* healthcare: label medical images to detect disease or monitor patient health\n* environment: label satellite images to monitor deforestation, inform wildland management or detect wildfires\n* agriculture: label images of crops to monitor plant health or satellite images for land use monitoring \n* ecology: label images of animal or plant species to monitor wildlife populations or track endangered species\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(task=\"image-classification\")\n>>> preds = classifier(\n...     \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg\"\n... )\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"]} for pred in preds]\n>>> print(*preds, sep=\"\\n\")\n{'score': 0.4335, 'label': 'lynx, catamount'}\n{'score': 0.0348, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}\n{'score': 0.0324, 'label': 'snow leopard, ounce, Panthera uncia'}\n{'score': 0.0239, 'label': 'Egyptian cat'}\n{'score': 0.0229, 'label': 'tiger cat'}\n```\n\n### Object detection\n\nUnlike image classification, object detection identifies multiple objects within an image and the objects' positions in an image (defined by the bounding box)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c864edd4-68f8-4d85-a3ba-775c8aacc867
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c864edd4-68f8-4d85-a3ba-775c8aacc867\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4ff5a904-2dd0-42c3-ab52-3cc4a5c03cf7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e3b26371b7397c4bed0c6457ab0a936cf202e2e15addb338512e81f6f599633b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b61feaf2-fcde-410f-ad2f-c0e1d3a69dd1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb7651ddc995abe45051847dfddef6855e5f108001faa70a06f9a62d6d23115e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4524, \"end_char_idx\": 8045, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "Split an image into patches and use a Transformer to gradually learn how each image patch is related to each other to form an image. Unlike the bottom-up approach favored by a CNN, this is kind of like starting out with a blurry image and then gradually bringing it into focus.\n\n### Image classification\n\nImage classification labels an entire image from a predefined set of classes. Like most classification tasks, there are many practical use cases for image classification, some of which include:\n\n* healthcare: label medical images to detect disease or monitor patient health\n* environment: label satellite images to monitor deforestation, inform wildland management or detect wildfires\n* agriculture: label images of crops to monitor plant health or satellite images for land use monitoring \n* ecology: label images of animal or plant species to monitor wildlife populations or track endangered species\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(task=\"image-classification\")\n>>> preds = classifier(\n...     \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg\"\n... )\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"]} for pred in preds]\n>>> print(*preds, sep=\"\\n\")\n{'score': 0.4335, 'label': 'lynx, catamount'}\n{'score': 0.0348, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}\n{'score': 0.0324, 'label': 'snow leopard, ounce, Panthera uncia'}\n{'score': 0.0239, 'label': 'Egyptian cat'}\n{'score': 0.0229, 'label': 'tiger cat'}\n```\n\n### Object detection\n\nUnlike image classification, object detection identifies multiple objects within an image and the objects' positions in an image (defined by the bounding box). Some example applications of object detection include:\n\n* self-driving vehicles: detect everyday traffic objects such as other vehicles, pedestrians, and traffic lights\n* remote sensing: disaster monitoring, urban planning, and weather forecasting\n* defect detection: detect cracks or structural damage in buildings, and manufacturing defects\n\n```py\n>>> from transformers import pipeline\n\n>>> detector = pipeline(task=\"object-detection\")\n>>> preds = detector(\n...     \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg\"\n... )\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"], \"box\": pred[\"box\"]} for pred in preds]\n>>> preds\n[{'score': 0.9865,\n  'label': 'cat',\n  'box': {'xmin': 178, 'ymin': 154, 'xmax': 882, 'ymax': 598}}]\n```\n\n### Image segmentation\n\nImage segmentation is a pixel-level task that assigns every pixel in an image to a class. It differs from object detection, which uses bounding boxes to label and predict objects in an image because segmentation is more granular. Segmentation can detect objects at a pixel-level. There are several types of image segmentation:\n\n* instance segmentation: in addition to labeling the class of an object, it also labels each distinct instance of an object (\"dog-1\", \"dog-2\")\n* panoptic segmentation: a combination of semantic and instance segmentation; it labels each pixel with a semantic class **and** each distinct instance of an object\n\nSegmentation tasks are helpful in self-driving vehicles to create a pixel-level map of the world around them so they can navigate safely around pedestrians and other vehicles. It is also useful for medical imaging, where the task's finer granularity can help identify abnormal cells or organ features."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b61feaf2-fcde-410f-ad2f-c0e1d3a69dd1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b61feaf2-fcde-410f-ad2f-c0e1d3a69dd1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c864edd4-68f8-4d85-a3ba-775c8aacc867\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"54a2f4ef00d609ba4184951458019bc5c3d5d1e549af25fe7046e5ec03c6c544\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"05f25b90-8395-427d-b2fc-1feb309299b5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ee65d4d906f42f271bce8e33034bbd9f2632a73029434b0e9788e112ba4c8957\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6278, \"end_char_idx\": 9846, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "Some example applications of object detection include:\n\n* self-driving vehicles: detect everyday traffic objects such as other vehicles, pedestrians, and traffic lights\n* remote sensing: disaster monitoring, urban planning, and weather forecasting\n* defect detection: detect cracks or structural damage in buildings, and manufacturing defects\n\n```py\n>>> from transformers import pipeline\n\n>>> detector = pipeline(task=\"object-detection\")\n>>> preds = detector(\n...     \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg\"\n... )\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"], \"box\": pred[\"box\"]} for pred in preds]\n>>> preds\n[{'score': 0.9865,\n  'label': 'cat',\n  'box': {'xmin': 178, 'ymin': 154, 'xmax': 882, 'ymax': 598}}]\n```\n\n### Image segmentation\n\nImage segmentation is a pixel-level task that assigns every pixel in an image to a class. It differs from object detection, which uses bounding boxes to label and predict objects in an image because segmentation is more granular. Segmentation can detect objects at a pixel-level. There are several types of image segmentation:\n\n* instance segmentation: in addition to labeling the class of an object, it also labels each distinct instance of an object (\"dog-1\", \"dog-2\")\n* panoptic segmentation: a combination of semantic and instance segmentation; it labels each pixel with a semantic class **and** each distinct instance of an object\n\nSegmentation tasks are helpful in self-driving vehicles to create a pixel-level map of the world around them so they can navigate safely around pedestrians and other vehicles. It is also useful for medical imaging, where the task's finer granularity can help identify abnormal cells or organ features. Image segmentation can also be used in ecommerce to virtually try on clothes or create augmented reality experiences by overlaying objects in the real world through your camera.\n\n```py\n>>> from transformers import pipeline\n\n>>> segmenter = pipeline(task=\"image-segmentation\")\n>>> preds = segmenter(\n...     \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg\"\n... )\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"]} for pred in preds]\n>>> print(*preds, sep=\"\\n\")\n{'score': 0.9879, 'label': 'LABEL_184'}\n{'score': 0.9973, 'label': 'snow'}\n{'score': 0.9972, 'label': 'cat'}\n```\n\n### Depth estimation\n\nDepth estimation predicts the distance of each pixel in an image from the camera. This computer vision task is especially important for scene understanding and reconstruction. For example, in self-driving cars, vehicles need to understand how far objects like pedestrians, traffic signs, and other vehicles are to avoid obstacles and collisions. Depth information is also helpful for constructing 3D representations from 2D images and can be used to create high-quality 3D representations of biological structures or buildings.\n\nThere are two approaches to depth estimation:\n\n* stereo: depths are estimated by comparing two images of the same image from slightly different angles\n* monocular: depths are estimated from a single image\n\n```py\n>>> from transformers import pipeline\n\n>>> depth_estimator = pipeline(task=\"depth-estimation\")\n>>> preds = depth_estimator(\n...     \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg\"\n... )\n```\n\n## Natural language processing\n\nNLP tasks are among the most common types of tasks because text is such a natural way for us to communicate."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 05f25b90-8395-427d-b2fc-1feb309299b5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"05f25b90-8395-427d-b2fc-1feb309299b5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b61feaf2-fcde-410f-ad2f-c0e1d3a69dd1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bf41c7dba5e3caa636cd4f5c375c68a0d98dc973c60b19ca9337fa1bf6a385a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"461ba8a0-4da8-49b2-bff0-0af11ca65434\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f01aa8c49f0bbe17fa282f2a40693c8394b6bf925081bc63da7356def6f64d05\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8046, \"end_char_idx\": 11630, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "Image segmentation can also be used in ecommerce to virtually try on clothes or create augmented reality experiences by overlaying objects in the real world through your camera.\n\n```py\n>>> from transformers import pipeline\n\n>>> segmenter = pipeline(task=\"image-segmentation\")\n>>> preds = segmenter(\n...     \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg\"\n... )\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"]} for pred in preds]\n>>> print(*preds, sep=\"\\n\")\n{'score': 0.9879, 'label': 'LABEL_184'}\n{'score': 0.9973, 'label': 'snow'}\n{'score': 0.9972, 'label': 'cat'}\n```\n\n### Depth estimation\n\nDepth estimation predicts the distance of each pixel in an image from the camera. This computer vision task is especially important for scene understanding and reconstruction. For example, in self-driving cars, vehicles need to understand how far objects like pedestrians, traffic signs, and other vehicles are to avoid obstacles and collisions. Depth information is also helpful for constructing 3D representations from 2D images and can be used to create high-quality 3D representations of biological structures or buildings.\n\nThere are two approaches to depth estimation:\n\n* stereo: depths are estimated by comparing two images of the same image from slightly different angles\n* monocular: depths are estimated from a single image\n\n```py\n>>> from transformers import pipeline\n\n>>> depth_estimator = pipeline(task=\"depth-estimation\")\n>>> preds = depth_estimator(\n...     \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg\"\n... )\n```\n\n## Natural language processing\n\nNLP tasks are among the most common types of tasks because text is such a natural way for us to communicate. To get text into a format recognized by a model, it needs to be tokenized. This means dividing a sequence of text into separate words or subwords (tokens) and then converting these tokens into numbers. As a result, you can represent a sequence of text as a sequence of numbers, and once you have a sequence of numbers, it can be input into a model to solve all sorts of NLP tasks!\n\n### Text classification\n\nLike classification tasks in any modality, text classification labels a sequence of text (it can be sentence-level, a paragraph, or a document) from a predefined set of classes. There are many practical applications for text classification, some of which include:\n\n* sentiment analysis: label text according to some polarity like `positive` or `negative` which can inform and support decision-making in fields like politics, finance, and marketing\n* content classification: label text according to some topic to help organize and filter information in news and social media feeds (`weather`, `sports`, `finance`, etc.)\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(task=\"sentiment-analysis\")\n>>> preds = classifier(\"Hugging Face is the best thing since sliced bread!\")\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"]} for pred in preds]\n>>> preds\n[{'score': 0.9991, 'label': 'POSITIVE'}]\n```\n\n### Token classification\n\nIn any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as [tokens](glossary#token). Token classification assigns each token a label from a predefined set of classes. \n\nTwo common types of token classification are:\n\n* named entity recognition (NER): label a token according to an entity category like organization, person, location or date."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 461ba8a0-4da8-49b2-bff0-0af11ca65434
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"461ba8a0-4da8-49b2-bff0-0af11ca65434\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"05f25b90-8395-427d-b2fc-1feb309299b5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"527efa3019b5c9c54efa55459975989184db1f2eea3ae00592a690b29705566e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"96c126ee-bba7-4229-8e80-bc01423634a0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dacb7c0bad5c895b8c6d63b6947868cef3eb2e897f90ce9e2402c4d66d752648\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9847, \"end_char_idx\": 12157, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "To get text into a format recognized by a model, it needs to be tokenized. This means dividing a sequence of text into separate words or subwords (tokens) and then converting these tokens into numbers. As a result, you can represent a sequence of text as a sequence of numbers, and once you have a sequence of numbers, it can be input into a model to solve all sorts of NLP tasks!\n\n### Text classification\n\nLike classification tasks in any modality, text classification labels a sequence of text (it can be sentence-level, a paragraph, or a document) from a predefined set of classes. There are many practical applications for text classification, some of which include:\n\n* sentiment analysis: label text according to some polarity like `positive` or `negative` which can inform and support decision-making in fields like politics, finance, and marketing\n* content classification: label text according to some topic to help organize and filter information in news and social media feeds (`weather`, `sports`, `finance`, etc.)\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(task=\"sentiment-analysis\")\n>>> preds = classifier(\"Hugging Face is the best thing since sliced bread!\")\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"]} for pred in preds]\n>>> preds\n[{'score': 0.9991, 'label': 'POSITIVE'}]\n```\n\n### Token classification\n\nIn any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as [tokens](glossary#token). Token classification assigns each token a label from a predefined set of classes. \n\nTwo common types of token classification are:\n\n* named entity recognition (NER): label a token according to an entity category like organization, person, location or date. NER is especially popular in biomedical settings, where it can label genes, proteins, and drug names.\n* part-of-speech tagging (POS): label a token according to its part-of-speech like noun, verb, or adjective. POS is useful for helping translation systems understand how two identical words are grammatically different (bank as a noun versus bank as a verb).\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(task=\"ner\")\n>>> preds = classifier(\"Hugging Face is a French company based in New York City.\")"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 96c126ee-bba7-4229-8e80-bc01423634a0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"96c126ee-bba7-4229-8e80-bc01423634a0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"461ba8a0-4da8-49b2-bff0-0af11ca65434\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"970d1e6e4dbda2b42cb19bfa0f039b5b6aefa50d500204ddba357a0ba50e0380\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5d1ce008-57a2-4902-a770-5915e8eeb5c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f29c07dad20ae21722adf83d6536ae294fb99f95b75a4a1d9e83061eaf2b83ec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10432, \"end_char_idx\": 13296, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "There are many practical applications for text classification, some of which include:\n\n* sentiment analysis: label text according to some polarity like `positive` or `negative` which can inform and support decision-making in fields like politics, finance, and marketing\n* content classification: label text according to some topic to help organize and filter information in news and social media feeds (`weather`, `sports`, `finance`, etc.)\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(task=\"sentiment-analysis\")\n>>> preds = classifier(\"Hugging Face is the best thing since sliced bread!\")\n>>> preds = [{\"score\": round(pred[\"score\"], 4), \"label\": pred[\"label\"]} for pred in preds]\n>>> preds\n[{'score': 0.9991, 'label': 'POSITIVE'}]\n```\n\n### Token classification\n\nIn any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as [tokens](glossary#token). Token classification assigns each token a label from a predefined set of classes. \n\nTwo common types of token classification are:\n\n* named entity recognition (NER): label a token according to an entity category like organization, person, location or date. NER is especially popular in biomedical settings, where it can label genes, proteins, and drug names.\n* part-of-speech tagging (POS): label a token according to its part-of-speech like noun, verb, or adjective. POS is useful for helping translation systems understand how two identical words are grammatically different (bank as a noun versus bank as a verb).\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(task=\"ner\")\n>>> preds = classifier(\"Hugging Face is a French company based in New York City.\")\n>>> preds = [\n...     {\n...         \"entity\": pred[\"entity\"],\n...         \"score\": round(pred[\"score\"], 4),\n...         \"index\": pred[\"index\"],\n...         \"word\": pred[\"word\"],\n...         \"start\": pred[\"start\"],\n...         \"end\": pred[\"end\"],\n...     }\n...     for pred in preds\n... ]\n>>> print(*preds, sep=\"\\n\")\n{'entity': 'I-ORG', 'score': 0.9968, 'index': 1, 'word': 'Hu', 'start': 0, 'end': 2}\n{'entity': 'I-ORG', 'score': 0.9293, 'index': 2, 'word': '##gging', 'start': 2, 'end': 7}\n{'entity': 'I-ORG', 'score': 0.9763, 'index': 3, 'word': 'Face', 'start': 8, 'end': 12}\n{'entity': 'I-MISC', 'score': 0.9983, 'index': 6, 'word': 'French', 'start': 18, 'end': 24}\n{'entity': 'I-LOC', 'score': 0.999, 'index': 10, 'word': 'New', 'start': 42, 'end': 45}\n{'entity': 'I-LOC', 'score': 0.9987, 'index': 11, 'word': 'York', 'start': 46, 'end': 50}\n{'entity': 'I-LOC', 'score': 0.9992, 'index': 12, 'word': 'City', 'start': 51, 'end': 55}\n```\n\n### Question answering\n\nQuestion answering is another token-level task that returns an answer to a question, sometimes with context (open-domain) and other times without context (closed-domain)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5d1ce008-57a2-4902-a770-5915e8eeb5c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5d1ce008-57a2-4902-a770-5915e8eeb5c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"96c126ee-bba7-4229-8e80-bc01423634a0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"662a057f46d7e3476debb5ddf37a77744730eaa129df10537f27353be64c6bbc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c944aede-5256-4448-b9a5-756f4c350880\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3e794fd959c9b63e7024ab64f859bce441a06f7db3ea2086fe79372522f0ecde\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13297, \"end_char_idx\": 16982, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "This task happens whenever we ask a virtual assistant something like whether a restaurant is open. It can also provide customer or technical support and help search engines retrieve the relevant information you're asking for. \n\nThere are two common types of question answering:\n\n* extractive: given a question and some context, the answer is a span of text from the context the model must extract\n* abstractive: given a question and some context, the answer is generated from the context; this approach is handled by the [`Text2TextGenerationPipeline`] instead of the [`QuestionAnsweringPipeline`] shown below\n\n\n```py\n>>> from transformers import pipeline\n\n>>> question_answerer = pipeline(task=\"question-answering\")\n>>> preds = question_answerer(\n...     question=\"What is the name of the repository?\",\n...     context=\"The name of the repository is huggingface/transformers\",\n... )\n>>> print(\n...     f\"score: {round(preds['score'], 4)}, start: {preds['start']}, end: {preds['end']}, answer: {preds['answer']}\"\n... )\nscore: 0.9327, start: 30, end: 54, answer: huggingface/transformers\n```\n\n### Summarization\n\nSummarization creates a shorter version of a text from a longer one while trying to preserve most of the meaning of the original document. Summarization is a sequence-to-sequence task; it outputs a shorter text sequence than the input. There are a lot of long-form documents that can be summarized to help readers quickly understand the main points. Legislative bills, legal and financial documents, patents, and scientific papers are a few examples of documents that could be summarized to save readers time and serve as a reading aid.\n\nLike question answering, there are two types of summarization:\n\n* extractive: identify and extract the most important sentences from the original text\n* abstractive: generate the target summary (which may include new words not in the input document) from the original text; the [`SummarizationPipeline`] uses the abstractive approach\n\n```py\n>>> from transformers import pipeline\n\n>>> summarizer = pipeline(task=\"summarization\")\n>>> summarizer(\n...     \"In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles.\"\n... )\n[{'summary_text': ' The Transformer is the first sequence transduction model based entirely on attention . It replaces the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention . For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers .'}]\n```\n\n### Translation\n\nTranslation converts a sequence of text in one language to another. It is important in helping people from different backgrounds communicate with each other, help translate content to reach wider audiences, and even be a learning tool to help people learn a new language. Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence. \n\nIn the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c944aede-5256-4448-b9a5-756f4c350880
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c944aede-5256-4448-b9a5-756f4c350880\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5d1ce008-57a2-4902-a770-5915e8eeb5c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a00de75ac4f3512ce364808aca0bcdc175b5652b26c21d546348ac9cd30e67fd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"86527754-ae04-4ff4-88cc-2617586d7aae\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7d418c436298dbea029ff6a7f74cd864b74efeb32e4b425332107fab58fdc26c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14946, \"end_char_idx\": 18535, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "Like question answering, there are two types of summarization:\n\n* extractive: identify and extract the most important sentences from the original text\n* abstractive: generate the target summary (which may include new words not in the input document) from the original text; the [`SummarizationPipeline`] uses the abstractive approach\n\n```py\n>>> from transformers import pipeline\n\n>>> summarizer = pipeline(task=\"summarization\")\n>>> summarizer(\n...     \"In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles.\"\n... )\n[{'summary_text': ' The Transformer is the first sequence transduction model based entirely on attention . It replaces the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention . For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers .'}]\n```\n\n### Translation\n\nTranslation converts a sequence of text in one language to another. It is important in helping people from different backgrounds communicate with each other, help translate content to reach wider audiences, and even be a learning tool to help people learn a new language. Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence. \n\nIn the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages.\n\n```py\n>>> from transformers import pipeline\n\n>>> text = \"translate English to French: Hugging Face is a community-based open-source platform for machine learning.\"\n>>> translator = pipeline(task=\"translation\", model=\"google-t5/t5-small\")\n>>> translator(text)\n[{'translation_text': \"Hugging Face est une tribune communautaire de l'apprentissage des machines.\"}]\n```\n\n### Language modeling\n\nLanguage modeling is a task that predicts a word in a sequence of text. It has become a very popular NLP task because a pretrained language model can be finetuned for many other downstream tasks. Lately, there has been a lot of interest in large language models (LLMs) which demonstrate zero- or few-shot learning. This means the model can solve tasks it wasn't explicitly trained to do! Language models can be used to generate fluent and convincing text, though you need to be careful since the text may not always be accurate.\n\nThere are two types of language modeling:\n\n* causal: the model's objective is to predict the next token in a sequence, and future tokens are masked\n\n    ```py\n    >>> from transformers import pipeline\n\n    >>> prompt = \"Hugging Face is a community-based open-source platform for machine learning.\"\n    >>> generator = pipeline(task=\"text-generation\")\n    >>> generator(prompt)  # doctest: +SKIP\n    ```\n\n* masked: the model's objective is to predict a masked token in a sequence with full access to the tokens in the sequence\n    \n    ```py\n    >>> text = \"Hugging Face is a community-based open-source <mask> for machine learning.\""}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 86527754-ae04-4ff4-88cc-2617586d7aae
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"86527754-ae04-4ff4-88cc-2617586d7aae\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c944aede-5256-4448-b9a5-756f4c350880\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"35c4896e47ed93ab7b068296a9a40139f63c80acb81a26879fe122c849d93b1c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6565f1a3-a6b3-4638-be6c-826a7ff4044f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0da49853a00474ed338686b9ccff22384b3dcbe7afb95a3a197d607cc3c1f83b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16641, \"end_char_idx\": 20295, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": "Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence. \n\nIn the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages.\n\n```py\n>>> from transformers import pipeline\n\n>>> text = \"translate English to French: Hugging Face is a community-based open-source platform for machine learning.\"\n>>> translator = pipeline(task=\"translation\", model=\"google-t5/t5-small\")\n>>> translator(text)\n[{'translation_text': \"Hugging Face est une tribune communautaire de l'apprentissage des machines.\"}]\n```\n\n### Language modeling\n\nLanguage modeling is a task that predicts a word in a sequence of text. It has become a very popular NLP task because a pretrained language model can be finetuned for many other downstream tasks. Lately, there has been a lot of interest in large language models (LLMs) which demonstrate zero- or few-shot learning. This means the model can solve tasks it wasn't explicitly trained to do! Language models can be used to generate fluent and convincing text, though you need to be careful since the text may not always be accurate.\n\nThere are two types of language modeling:\n\n* causal: the model's objective is to predict the next token in a sequence, and future tokens are masked\n\n    ```py\n    >>> from transformers import pipeline\n\n    >>> prompt = \"Hugging Face is a community-based open-source platform for machine learning.\"\n    >>> generator = pipeline(task=\"text-generation\")\n    >>> generator(prompt)  # doctest: +SKIP\n    ```\n\n* masked: the model's objective is to predict a masked token in a sequence with full access to the tokens in the sequence\n    \n    ```py\n    >>> text = \"Hugging Face is a community-based open-source <mask> for machine learning.\"\n    >>> fill_mask = pipeline(task=\"fill-mask\")\n    >>> preds = fill_mask(text, top_k=1)\n    >>> preds = [\n    ...     {\n    ...         \"score\": round(pred[\"score\"], 4),\n    ...         \"token\": pred[\"token\"],\n    ...         \"token_str\": pred[\"token_str\"],\n    ...         \"sequence\": pred[\"sequence\"],\n    ...     }\n    ...     for pred in preds\n    ... ]\n    >>> preds\n    [{'score': 0.2236,\n      'token': 1761,\n      'token_str': ' platform',\n      'sequence': 'Hugging Face is a community-based open-source platform for machine learning.'}]\n    ```\n\n## Multimodal\n\nMultimodal tasks require a model to process multiple data modalities (text, image, audio, video) to solve a particular problem. Image captioning is an example of a multimodal task where the model takes an image as input and outputs a sequence of text describing the image or some properties of the image. \n\nAlthough multimodal models work with different data types or modalities, internally, the preprocessing steps help the model convert all the data types into embeddings (vectors or list of numbers that holds meaningful information about the data). For a task like image captioning, the model learns relationships between image embeddings and text embeddings.\n\n### Document question answering\n\nDocument question answering is a task that answers natural language questions from a document. Unlike a token-level question answering task which takes text as input, document question answering takes an image of a document as input along with a question about the document and returns an answer. Document question answering can be used to parse structured documents and extract key information from it. In the example below, the total amount and change due can be extracted from a receipt."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6565f1a3-a6b3-4638-be6c-826a7ff4044f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6565f1a3-a6b3-4638-be6c-826a7ff4044f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82a98156-a531-459c-8251-4aa525ca11e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"86527754-ae04-4ff4-88cc-2617586d7aae\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/task_summary\", \"title\": \"What \\ud83e\\udd17 Transformers can do\", \"tokens\": 4858, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"54084d551ddb6e20e9b0c3b76e7deb11d59952c86b4cdf7ba8c5c1bf7ba8e4a0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18540, \"end_char_idx\": 21111, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82a98156-a531-459c-8251-4aa525ca11e3", "doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "ref_doc_id": "82a98156-a531-459c-8251-4aa525ca11e3", "chroma:document": ">>> fill_mask = pipeline(task=\"fill-mask\")\n    >>> preds = fill_mask(text, top_k=1)\n    >>> preds = [\n    ...     {\n    ...         \"score\": round(pred[\"score\"], 4),\n    ...         \"token\": pred[\"token\"],\n    ...         \"token_str\": pred[\"token_str\"],\n    ...         \"sequence\": pred[\"sequence\"],\n    ...     }\n    ...     for pred in preds\n    ... ]\n    >>> preds\n    [{'score': 0.2236,\n      'token': 1761,\n      'token_str': ' platform',\n      'sequence': 'Hugging Face is a community-based open-source platform for machine learning.'}]\n    ```\n\n## Multimodal\n\nMultimodal tasks require a model to process multiple data modalities (text, image, audio, video) to solve a particular problem. Image captioning is an example of a multimodal task where the model takes an image as input and outputs a sequence of text describing the image or some properties of the image. \n\nAlthough multimodal models work with different data types or modalities, internally, the preprocessing steps help the model convert all the data types into embeddings (vectors or list of numbers that holds meaningful information about the data). For a task like image captioning, the model learns relationships between image embeddings and text embeddings.\n\n### Document question answering\n\nDocument question answering is a task that answers natural language questions from a document. Unlike a token-level question answering task which takes text as input, document question answering takes an image of a document as input along with a question about the document and returns an answer. Document question answering can be used to parse structured documents and extract key information from it. In the example below, the total amount and change due can be extracted from a receipt.\n\n```py\n>>> from transformers import pipeline\n>>> from PIL import Image\n>>> import requests\n\n>>> url = \"https://huggingface.co/datasets/hf-internal-testing/example-documents/resolve/main/jpeg_images/2.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> doc_question_answerer = pipeline(\"document-question-answering\", model=\"magorshunov/layoutlm-invoices\")\n>>> preds = doc_question_answerer(\n...     question=\"What is the total amount?\",\n...     image=image,\n... )\n>>> preds\n[{'score': 0.8531, 'answer': '17,000', 'start': 4, 'end': 4}]\n```\n\nHopefully, this page has given you some more background information about all the types of tasks in each modality and the practical importance of each one. In the next [section](tasks_explained), you'll learn **how** \ud83e\udd17 Transformers work to solve these tasks."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b839ca8e-722c-44e5-ab1a-3085f9db1814
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b839ca8e-722c-44e5-ab1a-3085f9db1814\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/hpo_train\", \"title\": \"Hyperparameter Search using Trainer API\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1b5376d4-d342-4e26-8840-8ff69c7e9c4f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/hpo_train\", \"title\": \"Hyperparameter Search using Trainer API\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7d797771b1f9101ed3be1753409bd31a324784969f01bfec031a47be93a6c7d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bf22f72a-0050-459e-b9c1-eb361f51c590\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"017ffcb932fff7f132a0bcacc761fc07a8eedf1027a728bf51701e62a23b5d00\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2284, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "doc_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "ref_doc_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "chroma:document": "# Hyperparameter Search using Trainer API\n\n\ud83e\udd17 Transformers provides a [`Trainer`] class optimized for training \ud83e\udd17 Transformers models, making it easier to start training without manually writing your own training loop. The [`Trainer`] provides API for hyperparameter search. This doc shows how to enable it in example. \n\n## Hyperparameter Search backend\n\n[`Trainer`] supports four hyperparameter search backends currently:\n[optuna](https://optuna.org/), [sigopt](https://sigopt.com/), [raytune](https://docs.ray.io/en/latest/tune/index.html) and [wandb](https://wandb.ai/site/sweeps).\n\nyou should install them before using them as the hyperparameter search backend\n```bash\npip install optuna/sigopt/wandb/ray[tune] \n```\n\n## How to enable Hyperparameter search in example\n\nDefine the hyperparameter search space, different backends need different format.\n\nFor sigopt, see sigopt [object_parameter](https://docs.sigopt.com/ai-module-api-references/api_reference/objects/object_parameter), it's like following:\n```py\n>>> def sigopt_hp_space(trial):\n...     return [\n...         {\"bounds\": {\"min\": 1e-6, \"max\": 1e-4}, \"name\": \"learning_rate\", \"type\": \"double\"},\n...         {\n...             \"categorical_values\": [\"16\", \"32\", \"64\", \"128\"],\n...             \"name\": \"per_device_train_batch_size\",\n...             \"type\": \"categorical\",\n...         },\n...     ]\n```\n\nFor optuna, see optuna [object_parameter](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html#sphx-glr-tutorial-10-key-features-002-configurations-py), it's like following:\n\n```py\n>>> def optuna_hp_space(trial):\n...     return {\n...         \"learning_rate\": trial.suggest_float(\"learning_rate\", 1e-6, 1e-4, log=True),\n...         \"per_device_train_batch_size\": trial.suggest_categorical(\"per_device_train_batch_size\", [16, 32, 64, 128]),\n...     }\n```\n\nOptuna provides multi-objective HPO. You can pass `direction` in `hyperparameter_search` and define your own compute_objective to return multiple objective values. The Pareto Front (`List[BestRun]`) will be returned in hyperparameter_search, you should refer to the test case `TrainerHyperParameterMultiObjectOptunaIntegrationTest` in [test_trainer](https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: bf22f72a-0050-459e-b9c1-eb361f51c590
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bf22f72a-0050-459e-b9c1-eb361f51c590\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/hpo_train\", \"title\": \"Hyperparameter Search using Trainer API\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1b5376d4-d342-4e26-8840-8ff69c7e9c4f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/hpo_train\", \"title\": \"Hyperparameter Search using Trainer API\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7d797771b1f9101ed3be1753409bd31a324784969f01bfec031a47be93a6c7d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b839ca8e-722c-44e5-ab1a-3085f9db1814\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/hpo_train\", \"title\": \"Hyperparameter Search using Trainer API\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bed946cff9e56eca9bd01f48b6c4e14935230cf29fae29f715f4721e6c0ab747\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b52e1bf6-9fcf-45b3-b774-fd729884615f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8145890f6c49bce81fe9d1fb61705892269fbf1b8174c3093ed66350e7eead1a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 853, \"end_char_idx\": 4396, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "doc_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "ref_doc_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "chroma:document": "For sigopt, see sigopt [object_parameter](https://docs.sigopt.com/ai-module-api-references/api_reference/objects/object_parameter), it's like following:\n```py\n>>> def sigopt_hp_space(trial):\n...     return [\n...         {\"bounds\": {\"min\": 1e-6, \"max\": 1e-4}, \"name\": \"learning_rate\", \"type\": \"double\"},\n...         {\n...             \"categorical_values\": [\"16\", \"32\", \"64\", \"128\"],\n...             \"name\": \"per_device_train_batch_size\",\n...             \"type\": \"categorical\",\n...         },\n...     ]\n```\n\nFor optuna, see optuna [object_parameter](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html#sphx-glr-tutorial-10-key-features-002-configurations-py), it's like following:\n\n```py\n>>> def optuna_hp_space(trial):\n...     return {\n...         \"learning_rate\": trial.suggest_float(\"learning_rate\", 1e-6, 1e-4, log=True),\n...         \"per_device_train_batch_size\": trial.suggest_categorical(\"per_device_train_batch_size\", [16, 32, 64, 128]),\n...     }\n```\n\nOptuna provides multi-objective HPO. You can pass `direction` in `hyperparameter_search` and define your own compute_objective to return multiple objective values. The Pareto Front (`List[BestRun]`) will be returned in hyperparameter_search, you should refer to the test case `TrainerHyperParameterMultiObjectOptunaIntegrationTest` in [test_trainer](https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py). It's like following\n\n```py\n>>> best_trials = trainer.hyperparameter_search(\n...     direction=[\"minimize\", \"maximize\"],\n...     backend=\"optuna\",\n...     hp_space=optuna_hp_space,\n...     n_trials=20,\n...     compute_objective=compute_objective,\n... )\n```\n\nFor raytune, see raytune [object_parameter](https://docs.ray.io/en/latest/tune/api/search_space.html), it's like following:\n\n```py\n>>> def ray_hp_space(trial):\n...     return {\n...         \"learning_rate\": tune.loguniform(1e-6, 1e-4),\n...         \"per_device_train_batch_size\": tune.choice([16, 32, 64, 128]),\n...     }\n```\n\nFor wandb, see wandb [object_parameter](https://docs.wandb.ai/guides/sweeps/configuration), it's like following:\n\n```py\n>>> def wandb_hp_space(trial):\n...     return {\n...         \"method\": \"random\",\n...         \"metric\": {\"name\": \"objective\", \"goal\": \"minimize\"},\n...         \"parameters\": {\n...             \"learning_rate\": {\"distribution\": \"uniform\", \"min\": 1e-6, \"max\": 1e-4},\n...             \"per_device_train_batch_size\": {\"values\": [16, 32, 64, 128]},\n...         },\n...     }\n```\n\nDefine a `model_init` function and pass it to the [`Trainer`], as an example:\n```py\n>>> def model_init(trial):\n...     return AutoModelForSequenceClassification.from_pretrained(\n...         model_args.model_name_or_path,\n...         from_tf=bool(\".ckpt\" in model_args.model_name_or_path),\n...         config=config,\n...         cache_dir=model_args.cache_dir,\n...         revision=model_args.model_revision,\n...         token=True if model_args.use_auth_token else None,\n...     )\n```\n\nCreate a [`Trainer`] with your `model_init` function, training arguments, training and test datasets, and evaluation function:\n\n```py\n>>> trainer = Trainer(\n...     model=None,\n...     args=training_args,\n...     train_dataset=small_train_dataset,\n...     eval_dataset=small_eval_dataset,\n...     compute_metrics=compute_metrics,\n...     tokenizer=tokenizer,\n...     model_init=model_init,\n...     data_collator=data_collator,\n... )\n```\n\nCall hyperparameter search, get the best trial parameters, backend could be `\"optuna\"`/`\"sigopt\"`/`\"wandb\"`/`\"ray\"`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b52e1bf6-9fcf-45b3-b774-fd729884615f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b52e1bf6-9fcf-45b3-b774-fd729884615f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/hpo_train\", \"title\": \"Hyperparameter Search using Trainer API\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1b5376d4-d342-4e26-8840-8ff69c7e9c4f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/hpo_train\", \"title\": \"Hyperparameter Search using Trainer API\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7d797771b1f9101ed3be1753409bd31a324784969f01bfec031a47be93a6c7d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bf22f72a-0050-459e-b9c1-eb361f51c590\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/hpo_train\", \"title\": \"Hyperparameter Search using Trainer API\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fd4ac6bd272e6f7923fb7bc19e055aa2ed159d27d69db3cff828897d8f462865\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4397, \"end_char_idx\": 5125, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "doc_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "ref_doc_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "chroma:document": "direction can be`\"minimize\"` or `\"maximize\"`, which indicates whether to optimize greater or lower objective.\n\nYou could define your own compute_objective function, if not defined, the default compute_objective will be called, and the sum of eval metric like f1 is returned as objective value.\n\n```py\n>>> best_trial = trainer.hyperparameter_search(\n...     direction=\"maximize\",\n...     backend=\"optuna\",\n...     hp_space=optuna_hp_space,\n...     n_trials=20,\n...     compute_objective=compute_objective,\n... )\n```\n\n## Hyperparameter search For DDP finetune\nCurrently, Hyperparameter search for DDP is enabled for optuna and sigopt. Only the rank-zero process will generate the search trial and pass the argument to other ranks."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2e22c5e9-1bf6-4a1c-9509-01a2ebfa1202
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"2e22c5e9-1bf6-4a1c-9509-01a2ebfa1202\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"79df0831-2d78-4f1a-99f3-69fac4aa6ba4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"005e41af40faf253ff851d022474fb9fc54a4ed3782803c354845a565c76f040\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3330, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "# Efficient Training on Multiple GPUs\n\nIf training a model on a single GPU is too slow or if the model's weights do not fit in a single GPU's memory, transitioning \nto a multi-GPU setup may be a viable option. Prior to making this transition, thoroughly explore all the strategies covered \nin the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one) as they are universally applicable \nto model training on any number of GPUs. Once you have employed those strategies and found them insufficient for your \ncase on a single GPU, consider moving to multiple GPUs.\n\nTransitioning from a single GPU to multiple GPUs requires the introduction of some form of parallelism, as the workload \nmust be distributed across the resources. Multiple techniques can be employed to achieve parallelism, such as data \nparallelism, tensor parallelism, and pipeline parallelism. It's important to note that there isn't a one-size-fits-all \nsolution, and the optimal settings depend on the specific hardware configuration you are using. \n\nThis guide offers an in-depth overview of individual types of parallelism, as well as guidance on ways to combine   \ntechniques and choosing an appropriate approach. For step-by-step tutorials on distributed training, please refer to\nthe [\ud83e\udd17 Accelerate documentation](https://huggingface.co/docs/accelerate/index). \n\n<Tip>\n\nWhile the main concepts discussed in this guide are likely applicable across frameworks, here we focus on \nPyTorch-based implementations.\n\n</Tip>\n\nBefore diving deeper into the specifics of each technique, let's go over the rough decision process when training \nlarge models on a large infrastructure.\n\n## Scalability strategy\n\nBegin by estimating how much vRAM is required to train your model. For models hosted on the \ud83e\udd17 Hub, use our \n[Model Memory Calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage), which gives you \naccurate calculations within a few percent margin.  \n\n**Parallelization strategy for a single Node / multi-GPU setup**\n\nWhen training a model on a single node with multiple GPUs, your choice of parallelization strategy can significantly \nimpact performance. Here's a breakdown of your options:\n\n**Case 1: Your model fits onto a single GPU**\n\nIf your model can comfortably fit onto a single GPU, you have two primary options:\n\n1. DDP - Distributed DataParallel\n2. [Zero Redundancy Optimizer (ZeRO)](https://arxiv.org/abs/1910.02054) - depending on the situation and configuration used, this method may or may not be faster, however, it's worth experimenting with it.\n\n**Case 2: Your model doesn't fit onto a single GPU:**\n\nIf your model is too large for a single GPU, you have several alternatives to consider:\n\n1. PipelineParallel (PP)\n2. [ZeRO](https://arxiv.org/abs/1910.02054)\n3. [TensorParallel](#tensor-parallelism) (TP)\n\nWith very fast inter-node connectivity (e.g., NVLINK or NVSwitch) all three strategies (PP, ZeRO, TP) should result in \nsimilar performance. However, without these, PP will be faster than TP or ZeRO. The degree of TP may also \nmake a difference. It's best to experiment with your specific setup to determine the most suitable strategy.\n\nTP is almost always used within a single node. That is TP size <= GPUs per node.\n\n**Case 3: Largest layer of your model does not fit onto a single GPU**\n\n1."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 79df0831-2d78-4f1a-99f3-69fac4aa6ba4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"79df0831-2d78-4f1a-99f3-69fac4aa6ba4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2e22c5e9-1bf6-4a1c-9509-01a2ebfa1202\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"edda93c2c16079dafb7ff6b27bd0bcfd8e4265d353887ce53172e1b2412a8cdd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e4222c35-a1cd-4027-b488-bf4ef5a4280e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b48e0432309b056879d994ae6b0d5a17be52371a5d7e868384d54788ca932a74\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1761, \"end_char_idx\": 4875, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "For models hosted on the \ud83e\udd17 Hub, use our \n[Model Memory Calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage), which gives you \naccurate calculations within a few percent margin.  \n\n**Parallelization strategy for a single Node / multi-GPU setup**\n\nWhen training a model on a single node with multiple GPUs, your choice of parallelization strategy can significantly \nimpact performance. Here's a breakdown of your options:\n\n**Case 1: Your model fits onto a single GPU**\n\nIf your model can comfortably fit onto a single GPU, you have two primary options:\n\n1. DDP - Distributed DataParallel\n2. [Zero Redundancy Optimizer (ZeRO)](https://arxiv.org/abs/1910.02054) - depending on the situation and configuration used, this method may or may not be faster, however, it's worth experimenting with it.\n\n**Case 2: Your model doesn't fit onto a single GPU:**\n\nIf your model is too large for a single GPU, you have several alternatives to consider:\n\n1. PipelineParallel (PP)\n2. [ZeRO](https://arxiv.org/abs/1910.02054)\n3. [TensorParallel](#tensor-parallelism) (TP)\n\nWith very fast inter-node connectivity (e.g., NVLINK or NVSwitch) all three strategies (PP, ZeRO, TP) should result in \nsimilar performance. However, without these, PP will be faster than TP or ZeRO. The degree of TP may also \nmake a difference. It's best to experiment with your specific setup to determine the most suitable strategy.\n\nTP is almost always used within a single node. That is TP size <= GPUs per node.\n\n**Case 3: Largest layer of your model does not fit onto a single GPU**\n\n1. If you are not using ZeRO, you have to use TensorParallel (TP), because PipelineParallel (PP) alone won't be sufficient to accommodate the large layer.\n2. If you are using ZeRO, additionally adopt techniques from the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one).\n\n**Parallelization strategy for a multi-Node / multi-GPU setup**\n\n* When you have fast inter-node connectivity (e.g., NVLINK or NVSwitch) consider using one of these options:\n\n    1. ZeRO - as it requires close to no modifications to the model\n    2. A combination of PipelineParallel(PP) with TensorParallel(TP) and DataParallel(DP) - this approach will result in fewer communications, but requires significant changes to the model\n\n* When you have slow inter-node connectivity and still low on GPU memory:\n\n    1. Employ a combination of DataParallel(DP) with PipelineParallel(PP), TensorParallel(TP), and ZeRO.\n\nIn the following sections of this guide we dig deeper into how these different parallelism methods work.\n\n## Data Parallelism\n\nEven with only 2 GPUs, you can readily leverage the accelerated training capabilities offered by PyTorch's built-in features, \nsuch as `DataParallel` (DP) and `DistributedDataParallel` (DDP). Note that \n[PyTorch documentation](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html) recommends to prefer \n`DistributedDataParallel` (DDP) over `DataParallel` (DP) for multi-GPU training as it works for all models.\nLet's take a look at how these two methods work and what makes them different."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e4222c35-a1cd-4027-b488-bf4ef5a4280e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"e4222c35-a1cd-4027-b488-bf4ef5a4280e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"79df0831-2d78-4f1a-99f3-69fac4aa6ba4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"6b696a993028743669c1db7d105a69818b9a60ac643ccf3bb2bb192143b7a516\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"309298b5-445d-46bd-8b79-1604f0c25ead\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bf5757eb6a7e4fcfc7abc76d7017f285b7f99b1c77e398a6ae64f7d2a18eabe1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3174, \"end_char_idx\": 6470, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "TP is almost always used within a single node. That is TP size <= GPUs per node.\n\n**Case 3: Largest layer of your model does not fit onto a single GPU**\n\n1. If you are not using ZeRO, you have to use TensorParallel (TP), because PipelineParallel (PP) alone won't be sufficient to accommodate the large layer.\n2. If you are using ZeRO, additionally adopt techniques from the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one).\n\n**Parallelization strategy for a multi-Node / multi-GPU setup**\n\n* When you have fast inter-node connectivity (e.g., NVLINK or NVSwitch) consider using one of these options:\n\n    1. ZeRO - as it requires close to no modifications to the model\n    2. A combination of PipelineParallel(PP) with TensorParallel(TP) and DataParallel(DP) - this approach will result in fewer communications, but requires significant changes to the model\n\n* When you have slow inter-node connectivity and still low on GPU memory:\n\n    1. Employ a combination of DataParallel(DP) with PipelineParallel(PP), TensorParallel(TP), and ZeRO.\n\nIn the following sections of this guide we dig deeper into how these different parallelism methods work.\n\n## Data Parallelism\n\nEven with only 2 GPUs, you can readily leverage the accelerated training capabilities offered by PyTorch's built-in features, \nsuch as `DataParallel` (DP) and `DistributedDataParallel` (DDP). Note that \n[PyTorch documentation](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html) recommends to prefer \n`DistributedDataParallel` (DDP) over `DataParallel` (DP) for multi-GPU training as it works for all models.\nLet's take a look at how these two methods work and what makes them different.\n\n### DataParallel vs DistributedDataParallel\n\nTo understand the key differences in inter-GPU communication overhead between the two methods, let's review the processes per batch:\n\n[DDP](https://pytorch.org/docs/master/notes/ddp.html):\n\n- At the start time the main process replicates the model once from GPU 0 to the rest of GPUs\n- Then for each batch:\n   1. Each GPU directly consumes its mini-batch of data.\n   2. During `backward`, once the local gradients are ready, they are averaged across all processes.\n\n[DP](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html):\n\nFor each batch:\n   1. GPU 0 reads the batch of data and then sends a mini-batch to each GPU.\n   2. The up-to-date model is replicated from GPU 0 to each GPU. \n   3. `forward` is executed, and output from each GPU is sent to GPU 0 to compute the loss.\n   4. The loss is distributed from GPU 0 to all GPUs, and `backward` is run. \n   5. Gradients from each GPU are sent to GPU 0 and averaged. \n\nKey differences include:\n1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.\nDDP copies data using [torch.distributed](https://pytorch.org/docs/master/distributed.html), while DP copies data within \nthe process via Python threads (which introduces limitations associated with GIL). As a result, **`DistributedDataParallel` (DDP) is generally faster than `DataParallel` (DP)** unless you have slow GPU card inter-connectivity.\n2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 309298b5-445d-46bd-8b79-1604f0c25ead
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"309298b5-445d-46bd-8b79-1604f0c25ead\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4222c35-a1cd-4027-b488-bf4ef5a4280e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"b1a6356698a0e08d26b9c013e52556c6de384f6863d8d15fc894d10c3bd9240d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4a362dc7-85c9-435b-9d60-4e3656f0396d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bebf73d163113ba00616cde8f96a8aaaa5c45c1401609b5cd9e2b64cd8d04f06\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4877, \"end_char_idx\": 7273, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "### DataParallel vs DistributedDataParallel\n\nTo understand the key differences in inter-GPU communication overhead between the two methods, let's review the processes per batch:\n\n[DDP](https://pytorch.org/docs/master/notes/ddp.html):\n\n- At the start time the main process replicates the model once from GPU 0 to the rest of GPUs\n- Then for each batch:\n   1. Each GPU directly consumes its mini-batch of data.\n   2. During `backward`, once the local gradients are ready, they are averaged across all processes.\n\n[DP](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html):\n\nFor each batch:\n   1. GPU 0 reads the batch of data and then sends a mini-batch to each GPU.\n   2. The up-to-date model is replicated from GPU 0 to each GPU. \n   3. `forward` is executed, and output from each GPU is sent to GPU 0 to compute the loss.\n   4. The loss is distributed from GPU 0 to all GPUs, and `backward` is run. \n   5. Gradients from each GPU are sent to GPU 0 and averaged. \n\nKey differences include:\n1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.\nDDP copies data using [torch.distributed](https://pytorch.org/docs/master/distributed.html), while DP copies data within \nthe process via Python threads (which introduces limitations associated with GIL). As a result, **`DistributedDataParallel` (DDP) is generally faster than `DataParallel` (DP)** unless you have slow GPU card inter-connectivity.\n2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization. \n3. DDP supports distributed training across multiple machines, whereas DP does not.\n\nThis is not an exhaustive list of differences between DP and DDP, however, other nuances are out of scope of this guide.\nYou can get a deeper understanding of these methods by reading this [article](https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/).\n\nLet's illustrate the differences between DP and DDP with an experiment. We'll benchmark the differences between DP and \nDDP with an added context of NVLink presence:  \n\n* Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (`NV2` in `nvidia-smi topo -m`).\n* Software: `pytorch-1.8-to-be` + `cuda-11.0` / `transformers==4.3.0.dev0`.\n\nTo disable the NVLink feature on one of the benchmarks, we use `NCCL_P2P_DISABLE=1`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4a362dc7-85c9-435b-9d60-4e3656f0396d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"4a362dc7-85c9-435b-9d60-4e3656f0396d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"309298b5-445d-46bd-8b79-1604f0c25ead\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"47aca6886d2272ced149346b81b34c2dc8d661a9ae3ff4a99f6be451a032eab9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5e86d998-02a5-4c8b-bd31-46a3466ec2b4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"85b9fa7888aa9d5271446a1e78e454e8f04022fe55412e29990e292f952b5cdb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5719, \"end_char_idx\": 8920, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "4. The loss is distributed from GPU 0 to all GPUs, and `backward` is run. \n   5. Gradients from each GPU are sent to GPU 0 and averaged. \n\nKey differences include:\n1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.\nDDP copies data using [torch.distributed](https://pytorch.org/docs/master/distributed.html), while DP copies data within \nthe process via Python threads (which introduces limitations associated with GIL). As a result, **`DistributedDataParallel` (DDP) is generally faster than `DataParallel` (DP)** unless you have slow GPU card inter-connectivity.\n2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization. \n3. DDP supports distributed training across multiple machines, whereas DP does not.\n\nThis is not an exhaustive list of differences between DP and DDP, however, other nuances are out of scope of this guide.\nYou can get a deeper understanding of these methods by reading this [article](https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/).\n\nLet's illustrate the differences between DP and DDP with an experiment. We'll benchmark the differences between DP and \nDDP with an added context of NVLink presence:  \n\n* Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (`NV2` in `nvidia-smi topo -m`).\n* Software: `pytorch-1.8-to-be` + `cuda-11.0` / `transformers==4.3.0.dev0`.\n\nTo disable the NVLink feature on one of the benchmarks, we use `NCCL_P2P_DISABLE=1`. \n\nHere is the benchmarking code and outputs:\n\n**DP**\n\n```bash\nrm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \\\npython examples/pytorch/language-modeling/run_clm.py \\\n--model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \\\n--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200\n\n{'train_runtime': 110.5948, 'train_samples_per_second': 1.808, 'epoch': 0.69}\n```\n\n**DDP w/ NVlink**\n\n```bash\nrm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \\\ntorchrun --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \\\n--model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \\\n--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200\n\n{'train_runtime': 101.9003, 'train_samples_per_second': 1.963, 'epoch': 0.69}\n```\n\n**DDP w/o NVlink**\n\n```bash\nrm -r /tmp/test-clm; NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1 \\\ntorchrun --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \\\n--model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \\\n--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200\n\n{'train_runtime': 131.4367, 'train_samples_per_second': 1.522, 'epoch': 0.69}\n```\n\nHere are the same benchmarking results gathered in a table for convenience:\n\n| Type   | NVlink | Time |\n| :----- | -----  | ---: |\n| 2:DP   | Y      | 110s |\n| 2:DDP  | Y      | 101s |\n| 2:DDP  | N      | 131s |\n\nAs you can see, in this case DP is ~10% slower than DDP with NVlink, but ~15% faster than DDP without NVlink."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5e86d998-02a5-4c8b-bd31-46a3466ec2b4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"5e86d998-02a5-4c8b-bd31-46a3466ec2b4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4a362dc7-85c9-435b-9d60-4e3656f0396d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"5670acada383953423275afbd73606c9ab38bdce9ce386abe45c3f5b64939b8d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7cda6709-6a25-4eb9-9871-05c68be2550f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"378466ac7060365bbec651c1916cacb9d2cf1733d494ac3482c81a50a344fcaf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8921, \"end_char_idx\": 11691, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "The real difference will depend on how much data each GPU needs to sync with the others - the more there is to sync, \nthe more a slow link will impede the overall runtime.\n\n## ZeRO Data Parallelism\n\nZeRO-powered data parallelism (ZeRO-DP) is illustrated in the following diagram from this [blog post](https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/).\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-zero.png\" alt=\"DeepSpeed-Image-1\"/>\n </div>\n\nWhile it may appear complex, it is a very similar concept to `DataParallel` (DP). The difference is that instead of \nreplicating the full model parameters, gradients and optimizer states, each GPU stores only a slice of it. Then, at \nrun-time when the full layer parameters are needed just for the given layer, all GPUs synchronize to give each other \nparts that they miss.\n\nTo illustrate this idea, consider a simple model with 3 layers (La, Lb, and Lc), where each layer has 3 parameters. \nLayer La, for example, has weights a0, a1 and a2:\n\n```\nLa | Lb | Lc\n---|----|---\na0 | b0 | c0\na1 | b1 | c1\na2 | b2 | c2\n```\n\nIf we have 3 GPUs, ZeRO-DP splits the model onto 3 GPUs like so:\n\n```\nGPU0:\nLa | Lb | Lc\n---|----|---\na0 | b0 | c0\n\nGPU1:\nLa | Lb | Lc\n---|----|---\na1 | b1 | c1\n\nGPU2:\nLa | Lb | Lc\n---|----|---\na2 | b2 | c2\n```\n\nIn a way, this is the same horizontal slicing as tensor parallelism, as opposed to Vertical \nslicing, where one puts whole layer-groups on different GPUs. Now let's see how this works: \n\nEach of these GPUs will get the usual mini-batch as it works in DP:\n\n```\nx0 => GPU0\nx1 => GPU1\nx2 => GPU2\n```\n\nThe inputs are passed without modifications as if they would be processed by the original model.\n\nFirst, the inputs get to the layer `La`. What happens at this point?\n\nOn GPU0: the x0 mini-batch requires the a0, a1, a2 parameters to do its forward path through the layer, but the GPU0 has only a0. \nIt will get a1 from GPU1 and a2 from GPU2, bringing all the pieces of the model together.\n\nIn parallel, GPU1 gets another mini-batch - x1. GPU1 has the a1 parameter, but needs a0 and a2, so it gets those from GPU0 and GPU2.\nSame happens to GPU2 that gets the mini-batch x2. It gets a0 and a1 from GPU0 and GPU1.\n\nThis way each of the 3 GPUs gets the full tensors reconstructed and makes a forward pass with its own mini-batch.\nAs soon as the calculation is done, the data that is no longer needed gets dropped - it's only used during the calculation. \nThe reconstruction is done efficiently via a pre-fetch.\n\nThen the whole process is repeated for layer Lb, then Lc forward-wise, and then backward Lc -> Lb -> La."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7cda6709-6a25-4eb9-9871-05c68be2550f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"7cda6709-6a25-4eb9-9871-05c68be2550f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5e86d998-02a5-4c8b-bd31-46a3466ec2b4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"ba2ec91a20f64213ab43a86eb2bef4b4b7405e40a77cd46c93edc379845389b3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8c4a4ffe-2df7-45dc-b7fa-9b51540219ef\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"39be32304d025f8427f9a7fa59cceb63b5213d09d8f261d9ebdd90c787ccd727\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10536, \"end_char_idx\": 13444, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "Now let's see how this works: \n\nEach of these GPUs will get the usual mini-batch as it works in DP:\n\n```\nx0 => GPU0\nx1 => GPU1\nx2 => GPU2\n```\n\nThe inputs are passed without modifications as if they would be processed by the original model.\n\nFirst, the inputs get to the layer `La`. What happens at this point?\n\nOn GPU0: the x0 mini-batch requires the a0, a1, a2 parameters to do its forward path through the layer, but the GPU0 has only a0. \nIt will get a1 from GPU1 and a2 from GPU2, bringing all the pieces of the model together.\n\nIn parallel, GPU1 gets another mini-batch - x1. GPU1 has the a1 parameter, but needs a0 and a2, so it gets those from GPU0 and GPU2.\nSame happens to GPU2 that gets the mini-batch x2. It gets a0 and a1 from GPU0 and GPU1.\n\nThis way each of the 3 GPUs gets the full tensors reconstructed and makes a forward pass with its own mini-batch.\nAs soon as the calculation is done, the data that is no longer needed gets dropped - it's only used during the calculation. \nThe reconstruction is done efficiently via a pre-fetch.\n\nThen the whole process is repeated for layer Lb, then Lc forward-wise, and then backward Lc -> Lb -> La.\n\n<Tip>\n\nThis mechanism is similar to an efficient group backpacking strategy: person A carries the tent, person B carries the stove,\nand person C carries the axe. Each night they all share what they have with others and get from others what they don't have, \nand in the morning they pack up their allocated type of gear and continue on their way. This is what ZeRO DP/Sharded DDP is.\nCompare this strategy to the simple one where each person has to carry their own tent, stove and axe (similar to \nDataParallel (DP and DDP) in PyTorch), which would be far more inefficient. \n\n</Tip>\n\nWhile reading the literature on this topic you may encounter the following synonyms: Sharded, Partitioned.\nIf you pay close attention the way ZeRO partitions the model's weights - it looks very similar to tensor parallelism \nwhich will be discussed later. This is because it partitions/shards each layer's weights, unlike vertical model parallelism \nwhich is discussed next.\n\nImplementations:\n\n- [DeepSpeed](https://www.deepspeed.ai/tutorials/zero/) ZeRO-DP stages 1+2+3\n- [`Accelerate` integration](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed) \n- [`transformers` integration](main_classes/trainer#trainer-integrations)\n\n## From Naive Model Parallelism to Pipeline Parallelism\n\nTo explain Pipeline parallelism, we'll first look into Naive Model Parallelism (MP), also known as Vertical MP. This approach\ninvolves distributing groups of model layers across multiple GPUs by assigning specific layers to specific GPUs with `.to()`. \nAs data flows through these layers, it is moved to the same GPU as the layer, while the other layers remain untouched.\n\nWe refer to this Model parallelism as \"Vertical\" because of how models are typically visualized."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8c4a4ffe-2df7-45dc-b7fa-9b51540219ef
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"8c4a4ffe-2df7-45dc-b7fa-9b51540219ef\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7cda6709-6a25-4eb9-9871-05c68be2550f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f94c80639eb959d39985d655b1eaddf316029daf6266529a439f0e02ad64eb28\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1d932fcf-f854-410d-9398-43ae4b44a197\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"88288d70eac682fa5f9e040976555b7a9d20dc293bf1a83ffae53da30b85456a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11693, \"end_char_idx\": 14883, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "<Tip>\n\nThis mechanism is similar to an efficient group backpacking strategy: person A carries the tent, person B carries the stove,\nand person C carries the axe. Each night they all share what they have with others and get from others what they don't have, \nand in the morning they pack up their allocated type of gear and continue on their way. This is what ZeRO DP/Sharded DDP is.\nCompare this strategy to the simple one where each person has to carry their own tent, stove and axe (similar to \nDataParallel (DP and DDP) in PyTorch), which would be far more inefficient. \n\n</Tip>\n\nWhile reading the literature on this topic you may encounter the following synonyms: Sharded, Partitioned.\nIf you pay close attention the way ZeRO partitions the model's weights - it looks very similar to tensor parallelism \nwhich will be discussed later. This is because it partitions/shards each layer's weights, unlike vertical model parallelism \nwhich is discussed next.\n\nImplementations:\n\n- [DeepSpeed](https://www.deepspeed.ai/tutorials/zero/) ZeRO-DP stages 1+2+3\n- [`Accelerate` integration](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed) \n- [`transformers` integration](main_classes/trainer#trainer-integrations)\n\n## From Naive Model Parallelism to Pipeline Parallelism\n\nTo explain Pipeline parallelism, we'll first look into Naive Model Parallelism (MP), also known as Vertical MP. This approach\ninvolves distributing groups of model layers across multiple GPUs by assigning specific layers to specific GPUs with `.to()`. \nAs data flows through these layers, it is moved to the same GPU as the layer, while the other layers remain untouched.\n\nWe refer to this Model parallelism as \"Vertical\" because of how models are typically visualized. For example, the \nfollowing diagram shows an 8-layer model split vertically into two slices, placing layers 0-3 onto \nGPU0 and 4-7 to GPU1:\n\n```\n================\n| Layer |      |\n|   0   |      |\n|   1   | GPU0 |\n|   2   |      |\n|   3   |      |\n================\n| Layer |      |\n|   4   |      |\n|   5   | GPU1 |\n|   6   |      |\n|   7   |      |\n================\n```\n\nIn this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data \nfrom layer 3 to 4 requires moving it from GPU0 to GPU1, introducing a communication overhead. If the participating \nGPUs are on the same compute node (e.g. same physical machine) this copying is fast, but if the GPUs are distributed \nacross different compute nodes (e.g. multiple machines), the communication overhead could be substantially greater.\n\nFollowing that, layers 4 to 7 work as they would in the original model. Upon completion of the 7th layer, there is often \na need to send the data back to layer 0 where the labels are (or alternatively send the labels to the last layer). Now the loss can be \ncomputed and the optimizer can do its work.\n\nNaive Model Parallelism comes several shortcomings:\n- **All but one GPU are idle at any given moment**: if 4 GPUs are used, it's nearly identical to quadrupling the amount of memory of a single GPU, and ignoring the rest of the hardware. \n- **Overhead in data transfer between devices**:  E.g."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1d932fcf-f854-410d-9398-43ae4b44a197
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"1d932fcf-f854-410d-9398-43ae4b44a197\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8c4a4ffe-2df7-45dc-b7fa-9b51540219ef\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"23084d12f13ed7910f74aca49a5628c97634c087c9fbab4687d037a02bd4e8cf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4bad0ca4-3813-4657-95f0-d16fd5fe3b6b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f21456a3a798fa9cd866e2e8e3633b38883697b0ac667f86952cf9ab85ef6a51\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13348, \"end_char_idx\": 16360, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "We refer to this Model parallelism as \"Vertical\" because of how models are typically visualized. For example, the \nfollowing diagram shows an 8-layer model split vertically into two slices, placing layers 0-3 onto \nGPU0 and 4-7 to GPU1:\n\n```\n================\n| Layer |      |\n|   0   |      |\n|   1   | GPU0 |\n|   2   |      |\n|   3   |      |\n================\n| Layer |      |\n|   4   |      |\n|   5   | GPU1 |\n|   6   |      |\n|   7   |      |\n================\n```\n\nIn this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data \nfrom layer 3 to 4 requires moving it from GPU0 to GPU1, introducing a communication overhead. If the participating \nGPUs are on the same compute node (e.g. same physical machine) this copying is fast, but if the GPUs are distributed \nacross different compute nodes (e.g. multiple machines), the communication overhead could be substantially greater.\n\nFollowing that, layers 4 to 7 work as they would in the original model. Upon completion of the 7th layer, there is often \na need to send the data back to layer 0 where the labels are (or alternatively send the labels to the last layer). Now the loss can be \ncomputed and the optimizer can do its work.\n\nNaive Model Parallelism comes several shortcomings:\n- **All but one GPU are idle at any given moment**: if 4 GPUs are used, it's nearly identical to quadrupling the amount of memory of a single GPU, and ignoring the rest of the hardware. \n- **Overhead in data transfer between devices**:  E.g. 4x 6GB cards will be able to accommodate the same size as 1x 24GB card using naive MP, but a single 24GB card will complete the training faster, because it doesn't have the data copying overhead. But, say, if you have 40GB cards and need to fit a 45GB model you can with 4x 40GB cards (but barely because of the gradient and optimizer states)\n- **Copying shared embeddings**: Shared embeddings may need to get copied back and forth between GPUs.\n\nNow that you are familiar with how the naive approach to model parallelism works and its shortcomings, let's look at Pipeline Parallelism (PP).\nPP is almost identical to a naive MP, but it solves the GPU idling problem by chunking the incoming batch into micro-batches \nand artificially creating a pipeline, which allows different GPUs to concurrently participate in the computation process.\n\nThe following illustration from the [GPipe paper](https://ai.googleblog.com/2019/03/introducing-gpipe-open-source-library.html) \nshows the naive MP on the top, and PP on the bottom:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-gpipe-bubble.png\" alt=\"MP vs PP\"/>\n</div>\n\nAt the bottom of the diagram, you can observe that the Pipeline Parallelism (PP) approach minimizes the number of idle \nGPU zones, referred to as 'bubbles'. Both parts of the diagram show a parallelism level of degree 4, meaning that 4 GPUs \nare involved in the pipeline."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4bad0ca4-3813-4657-95f0-d16fd5fe3b6b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"4bad0ca4-3813-4657-95f0-d16fd5fe3b6b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1d932fcf-f854-410d-9398-43ae4b44a197\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"7620cd2a4f2305f994b83300dc7d579bcd44dfb60c72420bbb578e7e55c6336a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4eecdbe4-c5db-4e97-aed2-f92a1be5af4e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f6b40adaaee90b28b6ff7e08d62b460cbe23ae996936bef9639f66f0631b33b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14829, \"end_char_idx\": 17787, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "- **Overhead in data transfer between devices**:  E.g. 4x 6GB cards will be able to accommodate the same size as 1x 24GB card using naive MP, but a single 24GB card will complete the training faster, because it doesn't have the data copying overhead. But, say, if you have 40GB cards and need to fit a 45GB model you can with 4x 40GB cards (but barely because of the gradient and optimizer states)\n- **Copying shared embeddings**: Shared embeddings may need to get copied back and forth between GPUs.\n\nNow that you are familiar with how the naive approach to model parallelism works and its shortcomings, let's look at Pipeline Parallelism (PP).\nPP is almost identical to a naive MP, but it solves the GPU idling problem by chunking the incoming batch into micro-batches \nand artificially creating a pipeline, which allows different GPUs to concurrently participate in the computation process.\n\nThe following illustration from the [GPipe paper](https://ai.googleblog.com/2019/03/introducing-gpipe-open-source-library.html) \nshows the naive MP on the top, and PP on the bottom:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-gpipe-bubble.png\" alt=\"MP vs PP\"/>\n</div>\n\nAt the bottom of the diagram, you can observe that the Pipeline Parallelism (PP) approach minimizes the number of idle \nGPU zones, referred to as 'bubbles'. Both parts of the diagram show a parallelism level of degree 4, meaning that 4 GPUs \nare involved in the pipeline. You can see that there's a forward path of 4 pipe stages (F0, F1, F2 and F3) followed by \na backward path in reverse order (B3, B2, B1, and B0).\n\nPP introduces a new hyperparameter to tune - `chunks`, which determines how many data chunks are sent in a sequence \nthrough the same pipe stage. For example, in the bottom diagram you can see `chunks=4`. GPU0 performs the same \nforward path on chunk 0, 1, 2 and 3 (F0,0, F0,1, F0,2, F0,3) and then it waits for other GPUs to do complete their work. \nOnly when the other GPUs begin to complete their work, GPU0 starts to work again doing the backward path for chunks \n3, 2, 1 and 0 (B0,3, B0,2, B0,1, B0,0).\n\nNote that this is the same concept as gradient accumulation steps. PyTorch uses `chunks`, while DeepSpeed refers \nto the same hyperparameter as gradient accumulation steps.\n\nBecause of the chunks, PP introduces the notion of micro-batches (MBS). DP splits the global data batch size into \nmini-batches, so if you have a DP degree of 4, a global batch size of 1024 gets split up into 4 mini-batches of \n256 each (1024/4). And if the number of `chunks` (or GAS) is 32 we end up with a micro-batch size of 8 (256/32). Each \nPipeline stage works with a single micro-batch at a time. To calculate the global batch size of the DP + PP setup, \nuse the formula: `mbs * chunks * dp_degree` (`8 * 32 * 4 = 1024`).\nWith `chunks=1` you end up with the naive MP, which is inefficient."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4eecdbe4-c5db-4e97-aed2-f92a1be5af4e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"4eecdbe4-c5db-4e97-aed2-f92a1be5af4e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4bad0ca4-3813-4657-95f0-d16fd5fe3b6b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"3e906ec450eeb649c122789b632a3e5695c37c4b96cac42b54dd3c4635e0a53f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9a025a62-d1f5-4717-a9d3-b4c8879f7763\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4bae61f592bceaec4da24c3c774a5e8315cba854e4f1fb2708c619d07aef88d2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16507, \"end_char_idx\": 19513, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "PP introduces a new hyperparameter to tune - `chunks`, which determines how many data chunks are sent in a sequence \nthrough the same pipe stage. For example, in the bottom diagram you can see `chunks=4`. GPU0 performs the same \nforward path on chunk 0, 1, 2 and 3 (F0,0, F0,1, F0,2, F0,3) and then it waits for other GPUs to do complete their work. \nOnly when the other GPUs begin to complete their work, GPU0 starts to work again doing the backward path for chunks \n3, 2, 1 and 0 (B0,3, B0,2, B0,1, B0,0).\n\nNote that this is the same concept as gradient accumulation steps. PyTorch uses `chunks`, while DeepSpeed refers \nto the same hyperparameter as gradient accumulation steps.\n\nBecause of the chunks, PP introduces the notion of micro-batches (MBS). DP splits the global data batch size into \nmini-batches, so if you have a DP degree of 4, a global batch size of 1024 gets split up into 4 mini-batches of \n256 each (1024/4). And if the number of `chunks` (or GAS) is 32 we end up with a micro-batch size of 8 (256/32). Each \nPipeline stage works with a single micro-batch at a time. To calculate the global batch size of the DP + PP setup, \nuse the formula: `mbs * chunks * dp_degree` (`8 * 32 * 4 = 1024`).\nWith `chunks=1` you end up with the naive MP, which is inefficient. With a large `chunks` value you end up with \ntiny micro-batch sizes which is also inefficient. For this reason, we encourage to experiment with the `chunks` value to \nfind the one that leads to the most efficient GPUs utilization.\n\nYou may notice a bubble of \"dead\" time on the diagram that can't be parallelized because the last `forward` stage \nhas to wait for `backward` to complete the pipeline. The purpose of finding the best value for `chunks` is to enable a high \nconcurrent GPU utilization across all participating GPUs which translates to minimizing the size of the bubble.\n\nPipeline API solutions have been implemented in:\n- PyTorch\n- DeepSpeed\n- Megatron-LM\n\nThese come with some shortcomings:\n- They have to modify the model quite heavily, because Pipeline requires one to rewrite the normal flow of modules into a `nn.Sequential` sequence of the same, which may require changes to the design of the model.\n- Currently the Pipeline API is very restricted. If you had a bunch of Python variables being passed in the very first stage of the Pipeline, you will have to find a way around it. Currently, the pipeline interface requires either a single Tensor or a tuple of Tensors as the only input and output. These tensors must have a batch size as the very first dimension, since pipeline is going to chunk the mini batch into micro-batches. Possible improvements are being discussed here https://github.com/pytorch/pytorch/pull/50693\n- Conditional control flow at the level of pipe stages is not possible - e.g., Encoder-Decoder models like T5 require special workarounds to handle a conditional encoder stage.\n- They have to arrange each layer so that the output of one layer becomes an input to the other layer."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9a025a62-d1f5-4717-a9d3-b4c8879f7763
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"9a025a62-d1f5-4717-a9d3-b4c8879f7763\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4eecdbe4-c5db-4e97-aed2-f92a1be5af4e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"decb13d3622f891d4f86e59839e97be00432f168b82fdbbcd5e56ee7e98b7cdf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b740f795-1b07-405b-9f74-526d876991cd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c195026b5a467d8dff6cde01624b0c46f0b525bc16611e79a1a01f53ce005fc3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17720, \"end_char_idx\": 20849, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "With `chunks=1` you end up with the naive MP, which is inefficient. With a large `chunks` value you end up with \ntiny micro-batch sizes which is also inefficient. For this reason, we encourage to experiment with the `chunks` value to \nfind the one that leads to the most efficient GPUs utilization.\n\nYou may notice a bubble of \"dead\" time on the diagram that can't be parallelized because the last `forward` stage \nhas to wait for `backward` to complete the pipeline. The purpose of finding the best value for `chunks` is to enable a high \nconcurrent GPU utilization across all participating GPUs which translates to minimizing the size of the bubble.\n\nPipeline API solutions have been implemented in:\n- PyTorch\n- DeepSpeed\n- Megatron-LM\n\nThese come with some shortcomings:\n- They have to modify the model quite heavily, because Pipeline requires one to rewrite the normal flow of modules into a `nn.Sequential` sequence of the same, which may require changes to the design of the model.\n- Currently the Pipeline API is very restricted. If you had a bunch of Python variables being passed in the very first stage of the Pipeline, you will have to find a way around it. Currently, the pipeline interface requires either a single Tensor or a tuple of Tensors as the only input and output. These tensors must have a batch size as the very first dimension, since pipeline is going to chunk the mini batch into micro-batches. Possible improvements are being discussed here https://github.com/pytorch/pytorch/pull/50693\n- Conditional control flow at the level of pipe stages is not possible - e.g., Encoder-Decoder models like T5 require special workarounds to handle a conditional encoder stage.\n- They have to arrange each layer so that the output of one layer becomes an input to the other layer.\n\nMore recent solutions include:\n- Varuna\n- Sagemaker\n\nWe have not experimented with Varuna and SageMaker but their papers report that they have overcome the list of problems \nmentioned above and that they require smaller changes to the user's model.\n\nImplementations:\n- [PyTorch](https://pytorch.org/docs/stable/pipeline.html) (initial support in pytorch-1.8, and progressively getting improved in 1.9 and more so in 1.10). Some [examples](https://github.com/pytorch/pytorch/blob/master/benchmarks/distributed/pipeline/pipe.py)\n- [DeepSpeed](https://www.deepspeed.ai/tutorials/pipeline/)\n- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation - no API.\n- [Varuna](https://github.com/microsoft/varuna)\n- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.\n- [OSLO](https://github.com/tunib-ai/oslo) - this is implemented based on the Hugging Face Transformers.\n\n\ud83e\udd17 Transformers status: as of this writing none of the models supports full-PP. GPT2 and T5 models have naive MP support. \nThe main obstacle is being unable to convert the models to `nn.Sequential` and have all the inputs to be Tensors. This \nis because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b740f795-1b07-405b-9f74-526d876991cd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b740f795-1b07-405b-9f74-526d876991cd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9a025a62-d1f5-4717-a9d3-b4c8879f7763\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"a19f44e0e1a8d24879d641314db2a60742cff32f25c14c69eefbbfbdfd7cfe3e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aa469da7-90d1-40d5-81ee-6f2b60e0792f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7ba53e8b46a5a74f990f09411f006cad4726e6002acb19f0a2d65cf95280ab50\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19411, \"end_char_idx\": 22419, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "- They have to arrange each layer so that the output of one layer becomes an input to the other layer.\n\nMore recent solutions include:\n- Varuna\n- Sagemaker\n\nWe have not experimented with Varuna and SageMaker but their papers report that they have overcome the list of problems \nmentioned above and that they require smaller changes to the user's model.\n\nImplementations:\n- [PyTorch](https://pytorch.org/docs/stable/pipeline.html) (initial support in pytorch-1.8, and progressively getting improved in 1.9 and more so in 1.10). Some [examples](https://github.com/pytorch/pytorch/blob/master/benchmarks/distributed/pipeline/pipe.py)\n- [DeepSpeed](https://www.deepspeed.ai/tutorials/pipeline/)\n- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation - no API.\n- [Varuna](https://github.com/microsoft/varuna)\n- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.\n- [OSLO](https://github.com/tunib-ai/oslo) - this is implemented based on the Hugging Face Transformers.\n\n\ud83e\udd17 Transformers status: as of this writing none of the models supports full-PP. GPT2 and T5 models have naive MP support. \nThe main obstacle is being unable to convert the models to `nn.Sequential` and have all the inputs to be Tensors. This \nis because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.\n\nDeepSpeed and Megatron-LM integrations are available in [\ud83e\udd17 Accelerate](https://huggingface.co/docs/accelerate/main/en/usage_guides/deepspeed)\n\nOther approaches:\n\nDeepSpeed, Varuna and SageMaker use the concept of an [Interleaved Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-core-features.html)\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-sagemaker-interleaved-pipeline.png\" alt=\"Interleaved pipeline execution\"/>\n</div>\n\nHere the bubble (idle time) is further minimized by prioritizing backward passes. Varuna further attempts to improve the \nschedule by using simulations to discover the most efficient scheduling.\n\nOSLO has pipeline parallelism implementation based on the Transformers without `nn.Sequential` conversion.\n\n## Tensor Parallelism\n\nIn Tensor Parallelism, each GPU processes a slice of a tensor and only aggregates the full tensor for operations requiring it.\nTo describe this method, this section of the guide relies on the concepts and diagrams from the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) \npaper: [Efficient Large-Scale Language Model Training on GPU Clusters](https://arxiv.org/abs/2104.04473).\n\nThe main building block of any transformer is a fully connected `nn.Linear` followed by a nonlinear activation `GeLU`.\nThe dot dot-product part of it, following the Megatron's paper notation, can be written as `Y = GeLU(XA)`, where `X` is \nan input vector, `Y` is the output vector, and `A` is the weight matrix."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: aa469da7-90d1-40d5-81ee-6f2b60e0792f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"aa469da7-90d1-40d5-81ee-6f2b60e0792f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b740f795-1b07-405b-9f74-526d876991cd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"e069ddd5bab6d1640b7097a215d5dff5376b313b6458389bb95ebd931c53ba3e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ff262349-3e66-48f6-8fb2-bac27dd7faff\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f34bf66b9ce0fd630ec6e95cc081ef3e3b5570135166177ef323ed45223e8d17\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20699, \"end_char_idx\": 23832, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "This \nis because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.\n\nDeepSpeed and Megatron-LM integrations are available in [\ud83e\udd17 Accelerate](https://huggingface.co/docs/accelerate/main/en/usage_guides/deepspeed)\n\nOther approaches:\n\nDeepSpeed, Varuna and SageMaker use the concept of an [Interleaved Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-core-features.html)\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-sagemaker-interleaved-pipeline.png\" alt=\"Interleaved pipeline execution\"/>\n</div>\n\nHere the bubble (idle time) is further minimized by prioritizing backward passes. Varuna further attempts to improve the \nschedule by using simulations to discover the most efficient scheduling.\n\nOSLO has pipeline parallelism implementation based on the Transformers without `nn.Sequential` conversion.\n\n## Tensor Parallelism\n\nIn Tensor Parallelism, each GPU processes a slice of a tensor and only aggregates the full tensor for operations requiring it.\nTo describe this method, this section of the guide relies on the concepts and diagrams from the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) \npaper: [Efficient Large-Scale Language Model Training on GPU Clusters](https://arxiv.org/abs/2104.04473).\n\nThe main building block of any transformer is a fully connected `nn.Linear` followed by a nonlinear activation `GeLU`.\nThe dot dot-product part of it, following the Megatron's paper notation, can be written as `Y = GeLU(XA)`, where `X` is \nan input vector, `Y` is the output vector, and `A` is the weight matrix.\n\nIf we look at the computation in matrix form, you can see how the matrix multiplication can be split between multiple GPUs:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_gemm.png\" alt=\"Parallel GEMM\"/>\n</div>\n\nIf we split the weight matrix `A` column-wise across `N` GPUs and perform matrix multiplications `XA_1` through `XA_n` in parallel, \nthen we will end up with `N` output vectors `Y_1, Y_2, ..., Y_n` which can be fed into `GeLU` independently:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-independent-gelu.png\" alt=\"Independent GeLU\"/>\n</div>\n\nUsing this principle, we can update a multi-layer perceptron of arbitrary depth, without the need for any synchronization \nbetween GPUs until the very end, where we need to reconstruct the output vector from shards. The Megatron-LM paper authors \nprovide a helpful illustration for that:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_shard_processing.png\" alt=\"Parallel shard processing\"/>\n</div>\n\nParallelizing the multi-headed attention layers is even simpler, since they are already inherently parallel, due to having \nmultiple independent heads!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ff262349-3e66-48f6-8fb2-bac27dd7faff
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ff262349-3e66-48f6-8fb2-bac27dd7faff\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aa469da7-90d1-40d5-81ee-6f2b60e0792f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d97a9fe91b2e878decbce2bcfc0d7116f08be5854ec9f19898139f3fd62e049a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c9d2c899-a694-4e9a-b32a-c891f774c1ec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1579173a19c4dc268652b0e186f5fbd1bed89637831373171cd6839dfb99a8e1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22226, \"end_char_idx\": 25408, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "The dot dot-product part of it, following the Megatron's paper notation, can be written as `Y = GeLU(XA)`, where `X` is \nan input vector, `Y` is the output vector, and `A` is the weight matrix.\n\nIf we look at the computation in matrix form, you can see how the matrix multiplication can be split between multiple GPUs:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_gemm.png\" alt=\"Parallel GEMM\"/>\n</div>\n\nIf we split the weight matrix `A` column-wise across `N` GPUs and perform matrix multiplications `XA_1` through `XA_n` in parallel, \nthen we will end up with `N` output vectors `Y_1, Y_2, ..., Y_n` which can be fed into `GeLU` independently:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-independent-gelu.png\" alt=\"Independent GeLU\"/>\n</div>\n\nUsing this principle, we can update a multi-layer perceptron of arbitrary depth, without the need for any synchronization \nbetween GPUs until the very end, where we need to reconstruct the output vector from shards. The Megatron-LM paper authors \nprovide a helpful illustration for that:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_shard_processing.png\" alt=\"Parallel shard processing\"/>\n</div>\n\nParallelizing the multi-headed attention layers is even simpler, since they are already inherently parallel, due to having \nmultiple independent heads!\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_self_attention.png\" alt=\"Parallel self-attention\"/>\n</div>\n\nSpecial considerations: TP requires very fast network, and therefore it's not advisable to do TP across more than one node. \nPractically, if a node has 4 GPUs, the highest TP degree is therefore 4. If you need a TP degree of 8, you need to use\nnodes that have at least 8 GPUs.\n\nThis section is based on the original much more [detailed TP overview](https://github.com/huggingface/transformers/issues/10321#issuecomment-783543530).\nby [@anton-l](https://github.com/anton-l).\n\nAlternative names:\n- DeepSpeed calls it [tensor slicing](https://www.deepspeed.ai/training/#model-parallelism)\n\nImplementations:\n- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation, as it's very model-specific\n- [parallelformers](https://github.com/tunib-ai/parallelformers) (only inference at the moment)\n- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.\n- [OSLO](https://github.com/tunib-ai/oslo) has the tensor parallelism implementation based on the Transformers.\n\nSageMaker combines TP with DP for a more efficient processing.\n\n\ud83e\udd17 Transformers status:\n- core: not yet implemented in the core\n- but if you want inference [parallelformers](https://github.com/tunib-ai/parallelformers) provides this support for most of our models. So until this is implemented in the core you can use theirs."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c9d2c899-a694-4e9a-b32a-c891f774c1ec
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c9d2c899-a694-4e9a-b32a-c891f774c1ec\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ff262349-3e66-48f6-8fb2-bac27dd7faff\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8558903d67dcc07964f69de962ba029ef82aa05e163ed36a26844674b4a1c49a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"77049925-6721-4279-8dc1-fc55bb1a8fb4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9daaac50d1e5dab161b4257cd75157a8a59554015a1c7a96a3cc7e93ad80c212\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23834, \"end_char_idx\": 26536, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_self_attention.png\" alt=\"Parallel self-attention\"/>\n</div>\n\nSpecial considerations: TP requires very fast network, and therefore it's not advisable to do TP across more than one node. \nPractically, if a node has 4 GPUs, the highest TP degree is therefore 4. If you need a TP degree of 8, you need to use\nnodes that have at least 8 GPUs.\n\nThis section is based on the original much more [detailed TP overview](https://github.com/huggingface/transformers/issues/10321#issuecomment-783543530).\nby [@anton-l](https://github.com/anton-l).\n\nAlternative names:\n- DeepSpeed calls it [tensor slicing](https://www.deepspeed.ai/training/#model-parallelism)\n\nImplementations:\n- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation, as it's very model-specific\n- [parallelformers](https://github.com/tunib-ai/parallelformers) (only inference at the moment)\n- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.\n- [OSLO](https://github.com/tunib-ai/oslo) has the tensor parallelism implementation based on the Transformers.\n\nSageMaker combines TP with DP for a more efficient processing.\n\n\ud83e\udd17 Transformers status:\n- core: not yet implemented in the core\n- but if you want inference [parallelformers](https://github.com/tunib-ai/parallelformers) provides this support for most of our models. So until this is implemented in the core you can use theirs. And hopefully training mode will be supported too.\n- Deepspeed-Inference also supports our BERT, GPT-2, and GPT-Neo models in their super-fast CUDA-kernel-based inference mode, see more [here](https://www.deepspeed.ai/tutorials/inference-tutorial/)\n\n\ud83e\udd17 Accelerate integrates with [TP from Megatron-LM](https://huggingface.co/docs/accelerate/v0.23.0/en/usage_guides/megatron_lm).\n\n## Data Parallelism + Pipeline Parallelism\n\nThe following diagram from the DeepSpeed [pipeline tutorial](https://www.deepspeed.ai/tutorials/pipeline/) demonstrates \nhow one can combine DP with PP.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-zero-dp-pp.png\" alt=\"DP + PP-2d\"/>\n</div>\n\nHere it's important to see how DP rank 0 doesn't see GPU2 and DP rank 1 doesn't see GPU3. To DP there is just GPUs 0 \nand 1 where it feeds data as if there were just 2 GPUs. GPU0 \"secretly\" offloads some of its load to GPU2 using PP. \nAnd GPU1 does the same by enlisting GPU3 to its aid.\n\nSince each dimension requires at least 2 GPUs, here you'd need at least 4 GPUs."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 77049925-6721-4279-8dc1-fc55bb1a8fb4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"77049925-6721-4279-8dc1-fc55bb1a8fb4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c9d2c899-a694-4e9a-b32a-c891f774c1ec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"2402fb110c9e148b965ec754f2bbfbf9395c12d647fa6a39012b04b9e204fa3e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b52bed6b-8f37-407c-b82c-3864a9a7b942\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"df1b6aab547fc51aab921e9791cf770aab086fc4c079af09e1964ca4389510c9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25084, \"end_char_idx\": 27959, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "SageMaker combines TP with DP for a more efficient processing.\n\n\ud83e\udd17 Transformers status:\n- core: not yet implemented in the core\n- but if you want inference [parallelformers](https://github.com/tunib-ai/parallelformers) provides this support for most of our models. So until this is implemented in the core you can use theirs. And hopefully training mode will be supported too.\n- Deepspeed-Inference also supports our BERT, GPT-2, and GPT-Neo models in their super-fast CUDA-kernel-based inference mode, see more [here](https://www.deepspeed.ai/tutorials/inference-tutorial/)\n\n\ud83e\udd17 Accelerate integrates with [TP from Megatron-LM](https://huggingface.co/docs/accelerate/v0.23.0/en/usage_guides/megatron_lm).\n\n## Data Parallelism + Pipeline Parallelism\n\nThe following diagram from the DeepSpeed [pipeline tutorial](https://www.deepspeed.ai/tutorials/pipeline/) demonstrates \nhow one can combine DP with PP.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-zero-dp-pp.png\" alt=\"DP + PP-2d\"/>\n</div>\n\nHere it's important to see how DP rank 0 doesn't see GPU2 and DP rank 1 doesn't see GPU3. To DP there is just GPUs 0 \nand 1 where it feeds data as if there were just 2 GPUs. GPU0 \"secretly\" offloads some of its load to GPU2 using PP. \nAnd GPU1 does the same by enlisting GPU3 to its aid.\n\nSince each dimension requires at least 2 GPUs, here you'd need at least 4 GPUs.\n\nImplementations:\n- [DeepSpeed](https://github.com/microsoft/DeepSpeed)\n- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)\n- [Varuna](https://github.com/microsoft/varuna)\n- [SageMaker](https://arxiv.org/abs/2111.05972)\n- [OSLO](https://github.com/tunib-ai/oslo)\n\n\ud83e\udd17 Transformers status: not yet implemented\n\n## Data Parallelism + Pipeline Parallelism + Tensor Parallelism\n\nTo get an even more efficient training a 3D parallelism is used where PP is combined with TP and DP. This can be seen in the following diagram.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-deepspeed-3d.png\" alt=\"dp-pp-tp-3d\"/>\n</div>\n\nThis diagram is from a blog post [3D parallelism: Scaling to trillion-parameter models](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/), which is a good read as well.\n\nSince each dimension requires at least 2 GPUs, here you'd need at least 8 GPUs.\n\nImplementations:\n- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - DeepSpeed also includes an even more efficient DP, which they call ZeRO-DP.\n- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)\n- [Varuna](https://github.com/microsoft/varuna)\n- [SageMaker](https://arxiv.org/abs/2111.05972)\n- [OSLO](https://github.com/tunib-ai/oslo)\n\n\ud83e\udd17 Transformers status: not yet implemented, since we have no PP and TP."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b52bed6b-8f37-407c-b82c-3864a9a7b942
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b52bed6b-8f37-407c-b82c-3864a9a7b942\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"77049925-6721-4279-8dc1-fc55bb1a8fb4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"b5c7201c4d33f54ecb0953baf9e3efdd912a5a19406cec33fa72a6b55e059ff1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9826968b-2bee-4fbd-9ff3-e91d080fa9e3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"798e89309d62d4c9ddb2bb15f18157f28d13b2ac30c4b68483856224c3d5be9e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26538, \"end_char_idx\": 29515, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "Implementations:\n- [DeepSpeed](https://github.com/microsoft/DeepSpeed)\n- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)\n- [Varuna](https://github.com/microsoft/varuna)\n- [SageMaker](https://arxiv.org/abs/2111.05972)\n- [OSLO](https://github.com/tunib-ai/oslo)\n\n\ud83e\udd17 Transformers status: not yet implemented\n\n## Data Parallelism + Pipeline Parallelism + Tensor Parallelism\n\nTo get an even more efficient training a 3D parallelism is used where PP is combined with TP and DP. This can be seen in the following diagram.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-deepspeed-3d.png\" alt=\"dp-pp-tp-3d\"/>\n</div>\n\nThis diagram is from a blog post [3D parallelism: Scaling to trillion-parameter models](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/), which is a good read as well.\n\nSince each dimension requires at least 2 GPUs, here you'd need at least 8 GPUs.\n\nImplementations:\n- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - DeepSpeed also includes an even more efficient DP, which they call ZeRO-DP.\n- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)\n- [Varuna](https://github.com/microsoft/varuna)\n- [SageMaker](https://arxiv.org/abs/2111.05972)\n- [OSLO](https://github.com/tunib-ai/oslo)\n\n\ud83e\udd17 Transformers status: not yet implemented, since we have no PP and TP.\n\n## ZeRO Data Parallelism + Pipeline Parallelism + Tensor Parallelism\n\nOne of the main features of DeepSpeed is ZeRO, which is a super-scalable extension of DP. It has already been \ndiscussed in [ZeRO Data Parallelism](#zero-data-parallelism). Normally it's a standalone feature that doesn't require PP or TP. \nBut it can be combined with PP and TP.\n\nWhen ZeRO-DP is combined with PP (and optionally TP) it typically enables only ZeRO stage 1 (optimizer sharding).\n\nWhile it's theoretically possible to use ZeRO stage 2 (gradient sharding) with Pipeline Parallelism, it will have negative \nperformance impacts. There would need to be an additional reduce-scatter collective for every micro-batch to aggregate \nthe gradients before sharding, which adds a potentially significant communication overhead. By nature of Pipeline Parallelism, \nsmall micro-batches are used and instead the focus is on trying to balance arithmetic intensity (micro-batch size) with\nminimizing the Pipeline bubble (number of micro-batches). Therefore those communication costs are going to impact the performance.\n\nIn addition, there are already fewer layers than normal due to PP and so the memory savings won't be huge. PP already \nreduces gradient size by ``1/PP``, and so gradient sharding savings on top of that are less significant than pure DP.\n\nZeRO stage 3 is not a good choice either for the same reason - more inter-node communications required.\n\nAnd since we have ZeRO, the other benefit is ZeRO-Offload. Since this is stage 1 optimizer states can be offloaded to CPU."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9826968b-2bee-4fbd-9ff3-e91d080fa9e3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"9826968b-2bee-4fbd-9ff3-e91d080fa9e3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b52bed6b-8f37-407c-b82c-3864a9a7b942\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"7c5167ef9a5da1263d4449bcbb2850224245c015cb3dc0b7be9057afd754b2cf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"48a04f83-c359-4608-8401-dd7c1d8e9bbf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"44d101216853fa6795481b8a4b9604ed5895bc73e6c5de3023a821484adedad3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27961, \"end_char_idx\": 31196, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "## ZeRO Data Parallelism + Pipeline Parallelism + Tensor Parallelism\n\nOne of the main features of DeepSpeed is ZeRO, which is a super-scalable extension of DP. It has already been \ndiscussed in [ZeRO Data Parallelism](#zero-data-parallelism). Normally it's a standalone feature that doesn't require PP or TP. \nBut it can be combined with PP and TP.\n\nWhen ZeRO-DP is combined with PP (and optionally TP) it typically enables only ZeRO stage 1 (optimizer sharding).\n\nWhile it's theoretically possible to use ZeRO stage 2 (gradient sharding) with Pipeline Parallelism, it will have negative \nperformance impacts. There would need to be an additional reduce-scatter collective for every micro-batch to aggregate \nthe gradients before sharding, which adds a potentially significant communication overhead. By nature of Pipeline Parallelism, \nsmall micro-batches are used and instead the focus is on trying to balance arithmetic intensity (micro-batch size) with\nminimizing the Pipeline bubble (number of micro-batches). Therefore those communication costs are going to impact the performance.\n\nIn addition, there are already fewer layers than normal due to PP and so the memory savings won't be huge. PP already \nreduces gradient size by ``1/PP``, and so gradient sharding savings on top of that are less significant than pure DP.\n\nZeRO stage 3 is not a good choice either for the same reason - more inter-node communications required.\n\nAnd since we have ZeRO, the other benefit is ZeRO-Offload. Since this is stage 1 optimizer states can be offloaded to CPU.\n\nImplementations:\n- [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed) and [Megatron-Deepspeed from BigScience](https://github.com/bigscience-workshop/Megatron-DeepSpeed), which is the fork of the former repo.\n- [OSLO](https://github.com/tunib-ai/oslo)\n\nImportant papers:\n\n- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](\nhttps://arxiv.org/abs/2201.11990)\n\n\ud83e\udd17 Transformers status: not yet implemented, since we have no PP and TP.\n\n## FlexFlow\n\n[FlexFlow](https://github.com/flexflow/FlexFlow) also solves the parallelization problem in a slightly different approach.\n\nPaper: [\"Beyond Data and Model Parallelism for Deep Neural Networks\" by Zhihao Jia, Matei Zaharia, Alex Aiken](https://arxiv.org/abs/1807.05358)\n\nIt performs a sort of 4D Parallelism over Sample-Operator-Attribute-Parameter.\n\n1. Sample = Data Parallelism (sample-wise parallel)\n2. Operator = Parallelize a single operation into several sub-operations\n3. Attribute = Data Parallelism (length-wise parallel)\n4. Parameter = Model Parallelism (regardless of dimension - horizontal or vertical)\n\nExamples:\n* Sample\n\nLet's take 10 batches of sequence length 512. If we parallelize them by sample dimension into 2 devices, we get 10 x 512 which becomes be 5 x 2 x 512.\n\n* Operator\n\nIf we perform layer normalization, we compute std first and mean second, and then we can normalize data. \nOperator parallelism allows computing std and mean in parallel. So if we parallelize them by operator dimension into 2 \ndevices (cuda:0, cuda:1), first we copy input data into both devices, and cuda:0 computes std, cuda:1 computes mean at the same time."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 48a04f83-c359-4608-8401-dd7c1d8e9bbf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"48a04f83-c359-4608-8401-dd7c1d8e9bbf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9826968b-2bee-4fbd-9ff3-e91d080fa9e3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"ca19081580df4c369c168f64b64631d690664a08eba965819c02ca0febbef29a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ea6113b7-3194-4efb-ae96-ff85647df9d2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"464d90f681f30c534c12c07ead6a8428b748112b55decfe75eb4c0b554638191\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29745, \"end_char_idx\": 32980, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "- [OSLO](https://github.com/tunib-ai/oslo)\n\nImportant papers:\n\n- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](\nhttps://arxiv.org/abs/2201.11990)\n\n\ud83e\udd17 Transformers status: not yet implemented, since we have no PP and TP.\n\n## FlexFlow\n\n[FlexFlow](https://github.com/flexflow/FlexFlow) also solves the parallelization problem in a slightly different approach.\n\nPaper: [\"Beyond Data and Model Parallelism for Deep Neural Networks\" by Zhihao Jia, Matei Zaharia, Alex Aiken](https://arxiv.org/abs/1807.05358)\n\nIt performs a sort of 4D Parallelism over Sample-Operator-Attribute-Parameter.\n\n1. Sample = Data Parallelism (sample-wise parallel)\n2. Operator = Parallelize a single operation into several sub-operations\n3. Attribute = Data Parallelism (length-wise parallel)\n4. Parameter = Model Parallelism (regardless of dimension - horizontal or vertical)\n\nExamples:\n* Sample\n\nLet's take 10 batches of sequence length 512. If we parallelize them by sample dimension into 2 devices, we get 10 x 512 which becomes be 5 x 2 x 512.\n\n* Operator\n\nIf we perform layer normalization, we compute std first and mean second, and then we can normalize data. \nOperator parallelism allows computing std and mean in parallel. So if we parallelize them by operator dimension into 2 \ndevices (cuda:0, cuda:1), first we copy input data into both devices, and cuda:0 computes std, cuda:1 computes mean at the same time.\n\n* Attribute\n\nWe have 10 batches of 512 length. If we parallelize them by attribute dimension into 2 devices, 10 x 512 will be 10 x 2 x 256.\n\n* Parameter\n\nIt is similar with tensor model parallelism or naive layer-wise model parallelism.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-flexflow.jpeg\" alt=\"flex-flow-soap\"/>\n</div>\n\nThe significance of this framework is that it takes resources like (1) GPU/TPU/CPU vs. (2) RAM/DRAM vs. (3) \nfast-intra-connect/slow-inter-connect and it automatically optimizes all these algorithmically deciding which \nparallelisation to use where.\n\nOne very important aspect is that FlexFlow is designed for optimizing DNN parallelizations for models with static and \nfixed workloads, since models with dynamic behavior may prefer different parallelization strategies across iterations.\n\nSo the promise is very attractive - it runs a 30min simulation on the cluster of choice and it comes up with the best \nstrategy to utilise this specific environment. If you add/remove/replace any parts it'll run and re-optimize the plan \nfor that. And then you can train. A different setup will have its own custom optimization.\n\n\ud83e\udd17 Transformers status: Transformers models are FX-trace-able via [transformers.utils.fx](https://github.com/huggingface/transformers/blob/master/src/transformers/utils/fx.py), \nwhich is a prerequisite for FlexFlow, however, changes are required on the FlexFlow side to make it work with Transformers models.\n\n## GPU selection\n\nWhen training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ea6113b7-3194-4efb-ae96-ff85647df9d2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ea6113b7-3194-4efb-ae96-ff85647df9d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"48a04f83-c359-4608-8401-dd7c1d8e9bbf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"36e87959957cbc663eb512609fae1a0a1bdd8589b93c75eaf3a0cff0bc8a1589\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8896779a-3f18-425b-905b-208e5458bad9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ec9e668809547ed02db8f27a69bfa8e817b496b9f5b05eadebe77e201862218f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31245, \"end_char_idx\": 34248, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "If we parallelize them by attribute dimension into 2 devices, 10 x 512 will be 10 x 2 x 256.\n\n* Parameter\n\nIt is similar with tensor model parallelism or naive layer-wise model parallelism.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-flexflow.jpeg\" alt=\"flex-flow-soap\"/>\n</div>\n\nThe significance of this framework is that it takes resources like (1) GPU/TPU/CPU vs. (2) RAM/DRAM vs. (3) \nfast-intra-connect/slow-inter-connect and it automatically optimizes all these algorithmically deciding which \nparallelisation to use where.\n\nOne very important aspect is that FlexFlow is designed for optimizing DNN parallelizations for models with static and \nfixed workloads, since models with dynamic behavior may prefer different parallelization strategies across iterations.\n\nSo the promise is very attractive - it runs a 30min simulation on the cluster of choice and it comes up with the best \nstrategy to utilise this specific environment. If you add/remove/replace any parts it'll run and re-optimize the plan \nfor that. And then you can train. A different setup will have its own custom optimization.\n\n\ud83e\udd17 Transformers status: Transformers models are FX-trace-able via [transformers.utils.fx](https://github.com/huggingface/transformers/blob/master/src/transformers/utils/fx.py), \nwhich is a prerequisite for FlexFlow, however, changes are required on the FlexFlow side to make it work with Transformers models.\n\n## GPU selection\n\nWhen training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first. The selection process works for both [DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) and [DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html) to use only a subset of the available GPUs, and you don't need Accelerate or the [DeepSpeed integration](./main_classes/deepspeed).\n\n### Number of GPUs\n\nFor example, if you have 4 GPUs and you only want to use the first 2:\n\n<hfoptions id=\"select-gpu\">\n<hfoption id=\"torchrun\">\n\nUse the `--nproc_per_node` to select how many GPUs to use.\n\n```bash\ntorchrun --nproc_per_node=2  trainer-program.py ...\n```\n\n</hfoption>\n<hfoption id=\"Accelerate\">\n\nUse `--num_processes` to select how many GPUs to use.\n\n```bash\naccelerate launch --num_processes 2 trainer-program.py ...\n```\n\n</hfoption>\n<hfoption id=\"DeepSpeed\">\n\nUse `--num_gpus` to select how many GPUs to use.\n\n```bash\ndeepspeed --num_gpus 2 trainer-program.py ...\n```\n\n</hfoption>\n</hfoptions>\n\n### Order of GPUs\n\nNow, to select which GPUs to use and their order, you'll use the `CUDA_VISIBLE_DEVICES` environment variable. It is easiest to set the environment variable in a `~/bashrc` or another startup config file. `CUDA_VISIBLE_DEVICES` is used to map which GPUs are used."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8896779a-3f18-425b-905b-208e5458bad9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"8896779a-3f18-425b-905b-208e5458bad9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ea6113b7-3194-4efb-ae96-ff85647df9d2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"286fff1222eeb809215f173368f63d99883d85ab23e65ea2f9f517adb6a25428\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c7171c25-753f-467b-ba58-d4907c4da862\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d9887dcaa861f52c59264cf00383a0d11b00c25a3b1f749e63a7580a63931f94\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32750, \"end_char_idx\": 35762, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "## GPU selection\n\nWhen training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first. The selection process works for both [DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) and [DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html) to use only a subset of the available GPUs, and you don't need Accelerate or the [DeepSpeed integration](./main_classes/deepspeed).\n\n### Number of GPUs\n\nFor example, if you have 4 GPUs and you only want to use the first 2:\n\n<hfoptions id=\"select-gpu\">\n<hfoption id=\"torchrun\">\n\nUse the `--nproc_per_node` to select how many GPUs to use.\n\n```bash\ntorchrun --nproc_per_node=2  trainer-program.py ...\n```\n\n</hfoption>\n<hfoption id=\"Accelerate\">\n\nUse `--num_processes` to select how many GPUs to use.\n\n```bash\naccelerate launch --num_processes 2 trainer-program.py ...\n```\n\n</hfoption>\n<hfoption id=\"DeepSpeed\">\n\nUse `--num_gpus` to select how many GPUs to use.\n\n```bash\ndeepspeed --num_gpus 2 trainer-program.py ...\n```\n\n</hfoption>\n</hfoptions>\n\n### Order of GPUs\n\nNow, to select which GPUs to use and their order, you'll use the `CUDA_VISIBLE_DEVICES` environment variable. It is easiest to set the environment variable in a `~/bashrc` or another startup config file. `CUDA_VISIBLE_DEVICES` is used to map which GPUs are used. For example, if you have 4 GPUs (0, 1, 2, 3) and you only want to run GPUs 0 and 2:\n\n```bash\nCUDA_VISIBLE_DEVICES=0,2 torchrun trainer-program.py ...\n```\n\nOnly the 2 physical GPUs (0 and 2) are \"visible\" to PyTorch and these are mapped to `cuda:0` and `cuda:1` respectively. You can also reverse the order of the GPUs to use 2 first. Now, the mapping is `cuda:1` for GPU 0 and `cuda:0` for GPU 2.\n\n```bash\nCUDA_VISIBLE_DEVICES=2,0 torchrun trainer-program.py ...\n```\n\nYou can also set the `CUDA_VISIBLE_DEVICES` environment variable to an empty value to create an environment without GPUs.\n\n```bash\nCUDA_VISIBLE_DEVICES= python trainer-program.py ...\n```\n\n<Tip warning={true}>\n\nAs with any environment variable, they can be exported instead of being added to the command line. However, this is not recommended because it can be confusing if you forget how the environment variable was setup and you end up using the wrong GPUs. Instead, it is common practice to set the environment variable for a specific training run on the same command line.\n\n</Tip>\n\n`CUDA_DEVICE_ORDER` is an alternative environment variable you can use to control how the GPUs are ordered. You can either order them by:\n\n1. PCIe bus ID's that matches the order of [`nvidia-smi`](https://developer.nvidia.com/nvidia-system-management-interface) and [`rocm-smi`](https://rocm.docs.amd.com/projects/rocm_smi_lib/en/latest/.doxygen/docBin/html/index.html) for NVIDIA and AMD GPUs respectively\n\n```bash\nexport CUDA_DEVICE_ORDER=PCI_BUS_ID\n```\n\n2."}


