# embedding_metadata

key: chroma:document
string_value:
4.0/examples">v3.4.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.3.1/examples">v3.3.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.2.0/examples">v3.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.1.0/examples">v3.1.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.0.2/examples">v3.0.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.11.0/examples">v2.11.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.10.0/examples">v2.10.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.9.1/examples">v2.9.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.8.0/examples">v2.8.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.7.0/examples">v2.7.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.6.0/examples">v2.6.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.5.1/examples">v2.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.4.0/examples">v2.4.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.3.0/examples">v2.3.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.2.0/examples">v2.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.1.0/examples">v2.1.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.0.0/examples">v2.0.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.2.0/examples">v1.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.1.0/examples">v1.1.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.0.0/examples">v1.0.0</a></li>
	</ul>
</details>

Then switch your current clone of 🤗 Transformers to a specific version, like v3.5.1 for example:

```bash
git checkout tags/v3.5.


key: _node_content
string_value:
{"id_": "309349f8-bf4a-48ad-9ed9-f4443af3efd6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "090c97c6-5e83-4011-b620-c8db7549dd9c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "ad730df27f4daaf7a767eaa6caf66d60d1454262b6018432e636328ff29f5ceb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b318fe79-f45d-47cb-bbed-02e2bd736d92", "node_type": "1", "metadata": {}, "hash": "b4e9aefd194a9462fde45f20ed8dcc2f3017d3c3280fea6c45b2194f1bd8fa91", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4017, "end_char_idx": 6577, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0/examples">v2.6.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.5.1/examples">v2.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.4.0/examples">v2.4.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.3.0/examples">v2.3.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.2.0/examples">v2.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.1.0/examples">v2.1.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.0.0/examples">v2.0.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.2.0/examples">v1.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.1.0/examples">v1.1.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.0.0/examples">v1.0.0</a></li>
	</ul>
</details>

Then switch your current clone of 🤗 Transformers to a specific version, like v3.5.1 for example:

```bash
git checkout tags/v3.5.1
```

After you've setup the correct library version, navigate to the example folder of your choice and install the example specific requirements:

```bash
pip install -r requirements.txt
```

## Run a script

<frameworkcontent>
<pt>
The example script downloads and preprocesses a dataset from the 🤗 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset with the [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```
</pt>
<tf>
The example script downloads and preprocesses a dataset from the 🤗 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset using Keras on an architecture that supports summarization.


key: _node_content
string_value:
{"id_": "b318fe79-f45d-47cb-bbed-02e2bd736d92", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "309349f8-bf4a-48ad-9ed9-f4443af3efd6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "583d517e1cd8d0106ec8f41877fe11dc50a4ed0e83f77781c41da7ceb6aaa454", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c8bc21f0-9f9b-4af8-ab59-41d30d7b3ec4", "node_type": "1", "metadata": {}, "hash": "73402c8f930f349a1f30656c4a78f8fda594359a4fbe07a47e5cc1e6a12b55eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5019, "end_char_idx": 7666, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1 for example:

```bash
git checkout tags/v3.5.1
```

After you've setup the correct library version, navigate to the example folder of your choice and install the example specific requirements:

```bash
pip install -r requirements.txt
```

## Run a script

<frameworkcontent>
<pt>
The example script downloads and preprocesses a dataset from the 🤗 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset with the [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```
</pt>
<tf>
The example script downloads and preprocesses a dataset from the 🤗 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset using Keras on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.

```bash
python examples/tensorflow/summarization/run_summarization.py  \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval
```
</tf>
</frameworkcontent>

## Distributed training and mixed precision

The [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) supports distributed training and mixed precision, which means you can also use it in a script. To enable both of these features:

- Add the `fp16` argument to enable mixed precision.
- Set the number of GPUs to use with the `nproc_per_node` argument.


key: _node_content
string_value:
{"id_": "c8bc21f0-9f9b-4af8-ab59-41d30d7b3ec4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b318fe79-f45d-47cb-bbed-02e2bd736d92", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "8b20d92ea54ea32f1bae30f75f4e8c2d5a9a03c898335ecd704a294f4fd6d044", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5fb6703c-71fa-44fe-aa98-de3e1feb0074", "node_type": "1", "metadata": {}, "hash": "db016caf71e3c5138db8a105c02f04fe9b299a401e03dfc07de1d80fb4bb9f10", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6481, "end_char_idx": 9409, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then the script fine-tunes a dataset using Keras on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.

```bash
python examples/tensorflow/summarization/run_summarization.py  \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval
```
</tf>
</frameworkcontent>

## Distributed training and mixed precision

The [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) supports distributed training and mixed precision, which means you can also use it in a script. To enable both of these features:

- Add the `fp16` argument to enable mixed precision.
- Set the number of GPUs to use with the `nproc_per_node` argument.

```bash
torchrun \
    --nproc_per_node 8 pytorch/summarization/run_summarization.py \
    --fp16 \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```

TensorFlow scripts utilize a [`MirroredStrategy`](https://www.tensorflow.org/guide/distributed_training#mirroredstrategy) for distributed training, and you don't need to add any additional arguments to the training script. The TensorFlow script will use multiple GPUs by default if they are available.

## Run a script on a TPU

<frameworkcontent>
<pt>
Tensor Processing Units (TPUs) are specifically designed to accelerate performance. PyTorch supports TPUs with the [XLA](https://www.tensorflow.org/xla) deep learning compiler (see [here](https://github.com/pytorch/xla/blob/master/README.md) for more details). To use a TPU, launch the `xla_spawn.py` script and use the `num_cores` argument to set the number of TPU cores you want to use.

```bash
python xla_spawn.py --num_cores 8 \
    summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```
</pt>
<tf>
Tensor Processing Units (TPUs) are specifically designed to accelerate performance.


key: _node_content
string_value:
{"id_": "5fb6703c-71fa-44fe-aa98-de3e1feb0074", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c8bc21f0-9f9b-4af8-ab59-41d30d7b3ec4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "3b72191fc0217eca1a876ed10900f3f70cbdadcfe35805312af103bd7543e6d1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "969b074b-58f8-40d1-a9e9-9b94fdada796", "node_type": "1", "metadata": {}, "hash": "e054c4c38fb87a613a063b5b3a907454323311f6c90d085eda22374dcc27a79c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8351, "end_char_idx": 10750, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The TensorFlow script will use multiple GPUs by default if they are available.

## Run a script on a TPU

<frameworkcontent>
<pt>
Tensor Processing Units (TPUs) are specifically designed to accelerate performance. PyTorch supports TPUs with the [XLA](https://www.tensorflow.org/xla) deep learning compiler (see [here](https://github.com/pytorch/xla/blob/master/README.md) for more details). To use a TPU, launch the `xla_spawn.py` script and use the `num_cores` argument to set the number of TPU cores you want to use.

```bash
python xla_spawn.py --num_cores 8 \
    summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```
</pt>
<tf>
Tensor Processing Units (TPUs) are specifically designed to accelerate performance. TensorFlow scripts utilize a [`TPUStrategy`](https://www.tensorflow.org/guide/distributed_training#tpustrategy) for training on TPUs. To use a TPU, pass the name of the TPU resource to the `tpu` argument.

```bash
python run_summarization.py  \
    --tpu name_of_tpu_resource \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval
```
</tf>
</frameworkcontent>

## Run a script with 🤗 Accelerate

🤗 [Accelerate](https://huggingface.co/docs/accelerate) is a PyTorch-only library that offers a unified method for training a model on several types of setups (CPU-only, multiple GPUs, TPUs) while maintaining complete visibility into the PyTorch training loop. Make sure you have 🤗 Accelerate installed if you don't already have it:

> Note: As Accelerate is rapidly developing, the git version of accelerate must be installed to run the scripts
```bash
pip install git+https://github.com/huggingface/accelerate
```

Instead of the `run_summarization.py` script, you need to use the `run_summarization_no_trainer.py` script. 🤗 Accelerate supported scripts will have a `task_no_trainer.py` file in the folder.


key: _node_content
string_value:
{"id_": "969b074b-58f8-40d1-a9e9-9b94fdada796", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5fb6703c-71fa-44fe-aa98-de3e1feb0074", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "f8f88377319575b247ac2a1a62aa7c867257bd34947166dc471cee318ccafa82", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "616e0c54-1220-4c3c-aa71-d8c1561c3b10", "node_type": "1", "metadata": {}, "hash": "0c2ce3951a5b8f78bb76ac0d2b436d4947b0aaf83957d6d38a05272713026635", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9410, "end_char_idx": 12467, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
TensorFlow scripts utilize a [`TPUStrategy`](https://www.tensorflow.org/guide/distributed_training#tpustrategy) for training on TPUs. To use a TPU, pass the name of the TPU resource to the `tpu` argument.

```bash
python run_summarization.py  \
    --tpu name_of_tpu_resource \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval
```
</tf>
</frameworkcontent>

## Run a script with 🤗 Accelerate

🤗 [Accelerate](https://huggingface.co/docs/accelerate) is a PyTorch-only library that offers a unified method for training a model on several types of setups (CPU-only, multiple GPUs, TPUs) while maintaining complete visibility into the PyTorch training loop. Make sure you have 🤗 Accelerate installed if you don't already have it:

> Note: As Accelerate is rapidly developing, the git version of accelerate must be installed to run the scripts
```bash
pip install git+https://github.com/huggingface/accelerate
```

Instead of the `run_summarization.py` script, you need to use the `run_summarization_no_trainer.py` script. 🤗 Accelerate supported scripts will have a `task_no_trainer.py` file in the folder. Begin by running the following command to create and save a configuration file:

```bash
accelerate config
```

Test your setup to make sure it is configured correctly:

```bash
accelerate test
```

Now you are ready to launch the training:

```bash
accelerate launch run_summarization_no_trainer.py \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir ~/tmp/tst-summarization
```

## Use a custom dataset

The summarization script supports custom datasets as long as they are a CSV or JSON Line file. When you use your own dataset, you need to specify several additional arguments:

- `train_file` and `validation_file` specify the path to your training and validation files.
- `text_column` is the input text to summarize.
- `summary_column` is the target text to output.

A summarization script using a custom dataset would look like this:

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --train_file path_to_csv_or_jsonlines_file \
    --validation_file path_to_csv_or_jsonlines_file \
    --text_column text_column_name \
    --summary_column summary_column_name \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --overwrite_output_dir \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --predict_with_generate
```

## Test a script

It is often a good idea to run your script on a smaller number of dataset examples to ensure everything works as expected before committing to an entire dataset which may take hours to complete.


key: _node_content
string_value:
{"id_": "616e0c54-1220-4c3c-aa71-d8c1561c3b10", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "969b074b-58f8-40d1-a9e9-9b94fdada796", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "4b5962166f62f9e811c662ec755f47834e5930fc68e0fdd3d7c02cb664f96541", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dc0c442e-e484-4a4b-b03d-903937c0dc91", "node_type": "1", "metadata": {}, "hash": "785d9ac72f06f758f47f399e8d1fd78d291a871f9cbff4d221e7a83bc34cae0b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11367, "end_char_idx": 14432, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When you use your own dataset, you need to specify several additional arguments:

- `train_file` and `validation_file` specify the path to your training and validation files.
- `text_column` is the input text to summarize.
- `summary_column` is the target text to output.

A summarization script using a custom dataset would look like this:

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --train_file path_to_csv_or_jsonlines_file \
    --validation_file path_to_csv_or_jsonlines_file \
    --text_column text_column_name \
    --summary_column summary_column_name \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --overwrite_output_dir \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --predict_with_generate
```

## Test a script

It is often a good idea to run your script on a smaller number of dataset examples to ensure everything works as expected before committing to an entire dataset which may take hours to complete. Use the following arguments to truncate the dataset to a maximum number of samples:

- `max_train_samples`
- `max_eval_samples`
- `max_predict_samples`

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --max_train_samples 50 \
    --max_eval_samples 50 \
    --max_predict_samples 50 \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```

Not all example scripts support the `max_predict_samples` argument. If you aren't sure whether your script supports this argument, add the `-h` argument to check:

```bash
examples/pytorch/summarization/run_summarization.py -h
```

## Resume training from checkpoint

Another helpful option to enable is resuming training from a previous checkpoint. This will ensure you can pick up where you left off without starting over if your training gets interrupted. There are two methods to resume training from a checkpoint.

The first method uses the `output_dir previous_output_dir` argument to resume training from the latest checkpoint stored in `output_dir`. In this case, you should remove `overwrite_output_dir`:

```bash
python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --output_dir previous_output_dir \
    --predict_with_generate
```

The second method uses the `resume_from_checkpoint path_to_specific_checkpoint` argument to resume training from a specific checkpoint folder.


key: _node_content
string_value:
{"id_": "dc0c442e-e484-4a4b-b03d-903937c0dc91", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "616e0c54-1220-4c3c-aa71-d8c1561c3b10", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "acf508e4640aa74f398e33a35839ac4f96439ee4c80b1b58933d780ab99317dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13206, "end_char_idx": 16043, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you aren't sure whether your script supports this argument, add the `-h` argument to check:

```bash
examples/pytorch/summarization/run_summarization.py -h
```

## Resume training from checkpoint

Another helpful option to enable is resuming training from a previous checkpoint. This will ensure you can pick up where you left off without starting over if your training gets interrupted. There are two methods to resume training from a checkpoint.

The first method uses the `output_dir previous_output_dir` argument to resume training from the latest checkpoint stored in `output_dir`. In this case, you should remove `overwrite_output_dir`:

```bash
python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --output_dir previous_output_dir \
    --predict_with_generate
```

The second method uses the `resume_from_checkpoint path_to_specific_checkpoint` argument to resume training from a specific checkpoint folder.

```bash
python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --resume_from_checkpoint path_to_specific_checkpoint \
    --predict_with_generate
```

## Share your model

All scripts can upload your final model to the [Model Hub](https://huggingface.co/models). Make sure you are logged into Hugging Face before you begin:

```bash
huggingface-cli login
```

Then add the `push_to_hub` argument to the script. This argument will create a repository with your Hugging Face username and the folder name specified in `output_dir`.

To give your repository a specific name, use the `push_to_hub_model_id` argument to add it. The repository will be automatically listed under your namespace.

The following example shows how to upload a model with a specific repository name:

```bash
python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --push_to_hub \
    --push_to_hub_model_id finetuned-t5-cnn_dailymail \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```


key: _node_content
string_value:
{"id_": "cc980741-6a9e-4770-9198-ba9cff6ba928", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e6d92be0-7075-45f5-9f81-d8575684661e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "d4bfffcd0f7b9b6df805a1badab8bc6b6283a03c090d79361066e6a1f265c843", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "00dcd322-baa1-4031-bfb7-b0b4fb76fd63", "node_type": "1", "metadata": {}, "hash": "1b334b64c753c618cac8b4d6a5560427e8d1cdac3887a60e1c9a8b833de2d554", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Building custom models

The 🤗 Transformers library is designed to be easily extensible. Every model is fully coded in a given subfolder
of the repository with no abstraction, so you can easily copy a modeling file and tweak it to your needs.

If you are writing a brand new model, it might be easier to start from scratch. In this tutorial, we will show you
how to write a custom model and its configuration so it can be used inside Transformers, and how you can share it
with the community (with the code it relies on) so that anyone can use it, even if it's not present in the 🤗
Transformers library. We'll see how to build upon transformers and extend the framework with your hooks and
custom code.

We will illustrate all of this on a ResNet model, by wrapping the ResNet class of the
[timm library](https://github.com/rwightman/pytorch-image-models) into a [`PreTrainedModel`].

## Writing a custom configuration

Before we dive into the model, let's first write its configuration. The configuration of a model is an object that
will contain all the necessary information to build the model. As we will see in the next section, the model can only
take a `config` to be initialized, so we really need that object to be as complete as possible.

<Tip>

Models in the `transformers` library itself generally follow the convention that they accept a `config` object
in their `__init__` method, and then pass the whole `config` to sub-layers in the model, rather than breaking the 
config object into multiple arguments that are all passed individually to sub-layers. Writing your model in this 
style results in simpler code with a clear "source of truth" for any hyperparameters, and also makes it easier
to reuse code from other models in `transformers`.

</Tip>

In our example, we will take a couple of arguments of the ResNet class that we might want to tweak. Different
configurations will then give us the different types of ResNets that are possible. We then just store those arguments,
after checking the validity of a few of them.

```python
from transformers import PretrainedConfig
from typing import List


class ResnetConfig(PretrainedConfig):
    model_type = "resnet"

    def __init__(
        self,
        block_type="bottleneck",
        layers: List[int] = [3, 4, 6, 3],
        num_classes: int = 1000,
        input_channels: int = 3,
        cardinality: int = 1,
        base_width: int = 64,
        stem_width: int = 64,
        stem_type: str = "",
        avg_down: bool = False,
        **kwargs,
    ):
        if block_type not in ["basic", "bottleneck"]:
            raise ValueError(f"`block_type` must be 'basic' or bottleneck', got {block_type}.")
        if stem_type not in ["", "deep", "deep-tiered"]:
            raise ValueError(f"`stem_type` must be '', 'deep' or 'deep-tiered', got {stem_type}.")


key: _node_content
string_value:
{"id_": "00dcd322-baa1-4031-bfb7-b0b4fb76fd63", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e6d92be0-7075-45f5-9f81-d8575684661e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "d4bfffcd0f7b9b6df805a1badab8bc6b6283a03c090d79361066e6a1f265c843", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cc980741-6a9e-4770-9198-ba9cff6ba928", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "5ed88b9e577c7f8cc6bc12f9a09c5223dc9506347c089a1ff0731db2c1af16ee", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d39c977e-a08c-4512-b753-0a66947a848b", "node_type": "1", "metadata": {}, "hash": "4e25843629c9596925a099d680f7bd20bdcdb649c5dedad0acdab0788f43559e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2123, "end_char_idx": 5270, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class ResnetConfig(PretrainedConfig):
    model_type = "resnet"

    def __init__(
        self,
        block_type="bottleneck",
        layers: List[int] = [3, 4, 6, 3],
        num_classes: int = 1000,
        input_channels: int = 3,
        cardinality: int = 1,
        base_width: int = 64,
        stem_width: int = 64,
        stem_type: str = "",
        avg_down: bool = False,
        **kwargs,
    ):
        if block_type not in ["basic", "bottleneck"]:
            raise ValueError(f"`block_type` must be 'basic' or bottleneck', got {block_type}.")
        if stem_type not in ["", "deep", "deep-tiered"]:
            raise ValueError(f"`stem_type` must be '', 'deep' or 'deep-tiered', got {stem_type}.")

        self.block_type = block_type
        self.layers = layers
        self.num_classes = num_classes
        self.input_channels = input_channels
        self.cardinality = cardinality
        self.base_width = base_width
        self.stem_width = stem_width
        self.stem_type = stem_type
        self.avg_down = avg_down
        super().__init__(**kwargs)
```

The three important things to remember when writing you own configuration are the following:
- you have to inherit from `PretrainedConfig`,
- the `__init__` of your `PretrainedConfig` must accept any kwargs,
- those `kwargs` need to be passed to the superclass `__init__`.

The inheritance is to make sure you get all the functionality from the 🤗 Transformers library, while the two other
constraints come from the fact a `PretrainedConfig` has more fields than the ones you are setting. When reloading a
config with the `from_pretrained` method, those fields need to be accepted by your config and then sent to the
superclass.

Defining a `model_type` for your configuration (here `model_type="resnet"`) is not mandatory, unless you want to
register your model with the auto classes (see last section).

With this done, you can easily create and save your configuration like you would do with any other model config of the
library. Here is how we can create a resnet50d config and save it:

```py
resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d_config.save_pretrained("custom-resnet")
```

This will save a file named `config.json` inside the folder `custom-resnet`. You can then reload your config with the
`from_pretrained` method:

```py
resnet50d_config = ResnetConfig.from_pretrained("custom-resnet")
```

You can also use any other method of the [`PretrainedConfig`] class, like [`~PretrainedConfig.push_to_hub`] to
directly upload your config to the Hub.

## Writing a custom model

Now that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that
extracts the hidden features from a batch of images (like [`BertModel`]) and one that is suitable for image
classification (like [`BertForSequenceClassification`]).

As we mentioned before, we'll only write a loose wrapper of the model to keep it simple for this example. The only
thing we need to do before writing this class is a map between the block types and actual block classes.


key: _node_content
string_value:
{"id_": "d39c977e-a08c-4512-b753-0a66947a848b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e6d92be0-7075-45f5-9f81-d8575684661e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "d4bfffcd0f7b9b6df805a1badab8bc6b6283a03c090d79361066e6a1f265c843", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "00dcd322-baa1-4031-bfb7-b0b4fb76fd63", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "393b80bb1ee2086162a646c3fd770ce81cc83eb8f45194f983bac27de203f15e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d9b355ab-151b-48a2-b912-6b6a8c98bdd5", "node_type": "1", "metadata": {}, "hash": "22587b35b68bcb94166ab848b2769486eb813f9b4d497a75d56df26291ed4fea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3703, "end_char_idx": 6347, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When reloading a
config with the `from_pretrained` method, those fields need to be accepted by your config and then sent to the
superclass.

Defining a `model_type` for your configuration (here `model_type="resnet"`) is not mandatory, unless you want to
register your model with the auto classes (see last section).

With this done, you can easily create and save your configuration like you would do with any other model config of the
library. Here is how we can create a resnet50d config and save it:

```py
resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d_config.save_pretrained("custom-resnet")
```

This will save a file named `config.json` inside the folder `custom-resnet`. You can then reload your config with the
`from_pretrained` method:

```py
resnet50d_config = ResnetConfig.from_pretrained("custom-resnet")
```

You can also use any other method of the [`PretrainedConfig`] class, like [`~PretrainedConfig.push_to_hub`] to
directly upload your config to the Hub.

## Writing a custom model

Now that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that
extracts the hidden features from a batch of images (like [`BertModel`]) and one that is suitable for image
classification (like [`BertForSequenceClassification`]).

As we mentioned before, we'll only write a loose wrapper of the model to keep it simple for this example. The only
thing we need to do before writing this class is a map between the block types and actual block classes. Then the
model is defined from the configuration by passing everything to the `ResNet` class:

```py
from transformers import PreTrainedModel
from timm.models.resnet import BasicBlock, Bottleneck, ResNet
from .configuration_resnet import ResnetConfig


BLOCK_MAPPING = {"basic": BasicBlock, "bottleneck": Bottleneck}


class ResnetModel(PreTrainedModel):
    config_class = ResnetConfig

    def __init__(self, config):
        super().__init__(config)
        block_layer = BLOCK_MAPPING[config.block_type]
        self.model = ResNet(
            block_layer,
            config.layers,
            num_classes=config.num_classes,
            in_chans=config.input_channels,
            cardinality=config.cardinality,
            base_width=config.base_width,
            stem_width=config.stem_width,
            stem_type=config.stem_type,
            avg_down=config.avg_down,
        )

    def forward(self, tensor):
        return self.model.forward_features(tensor)
```

For the model that will classify images, we just change the forward method:

```py
import torch


key: _node_content
string_value:
{"id_": "d9b355ab-151b-48a2-b912-6b6a8c98bdd5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e6d92be0-7075-45f5-9f81-d8575684661e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "d4bfffcd0f7b9b6df805a1badab8bc6b6283a03c090d79361066e6a1f265c843", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d39c977e-a08c-4512-b753-0a66947a848b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "f221066e26c8badb62085abda26645d862794dd93424556804dc324cf44c0fc1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fd3e485b-6d00-4543-b4b3-737434b34037", "node_type": "1", "metadata": {}, "hash": "702567af40b8191685f370181cc253554298070bba59d092fc7a62c8ebfcae46", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4746, "end_char_idx": 8302, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Writing a custom model

Now that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that
extracts the hidden features from a batch of images (like [`BertModel`]) and one that is suitable for image
classification (like [`BertForSequenceClassification`]).

As we mentioned before, we'll only write a loose wrapper of the model to keep it simple for this example. The only
thing we need to do before writing this class is a map between the block types and actual block classes. Then the
model is defined from the configuration by passing everything to the `ResNet` class:

```py
from transformers import PreTrainedModel
from timm.models.resnet import BasicBlock, Bottleneck, ResNet
from .configuration_resnet import ResnetConfig


BLOCK_MAPPING = {"basic": BasicBlock, "bottleneck": Bottleneck}


class ResnetModel(PreTrainedModel):
    config_class = ResnetConfig

    def __init__(self, config):
        super().__init__(config)
        block_layer = BLOCK_MAPPING[config.block_type]
        self.model = ResNet(
            block_layer,
            config.layers,
            num_classes=config.num_classes,
            in_chans=config.input_channels,
            cardinality=config.cardinality,
            base_width=config.base_width,
            stem_width=config.stem_width,
            stem_type=config.stem_type,
            avg_down=config.avg_down,
        )

    def forward(self, tensor):
        return self.model.forward_features(tensor)
```

For the model that will classify images, we just change the forward method:

```py
import torch


class ResnetModelForImageClassification(PreTrainedModel):
    config_class = ResnetConfig

    def __init__(self, config):
        super().__init__(config)
        block_layer = BLOCK_MAPPING[config.block_type]
        self.model = ResNet(
            block_layer,
            config.layers,
            num_classes=config.num_classes,
            in_chans=config.input_channels,
            cardinality=config.cardinality,
            base_width=config.base_width,
            stem_width=config.stem_width,
            stem_type=config.stem_type,
            avg_down=config.avg_down,
        )

    def forward(self, tensor, labels=None):
        logits = self.model(tensor)
        if labels is not None:
            loss = torch.nn.functional.cross_entropy(logits, labels)
            return {"loss": loss, "logits": logits}
        return {"logits": logits}
```

In both cases, notice how we inherit from `PreTrainedModel` and call the superclass initialization with the `config`
(a bit like when you write a regular `torch.nn.Module`). The line that sets the `config_class` is not mandatory, unless
you want to register your model with the auto classes (see last section).

<Tip>

If your model is very similar to a model inside the library, you can re-use the same configuration as this model.

</Tip>

You can have your model return anything you want, but returning a dictionary like we did for
`ResnetModelForImageClassification`, with the loss included when labels are passed, will make your model directly
usable inside the [`Trainer`] class. Using another output format is fine as long as you are planning on using your own
training loop or another library for training.

Now that we have our model class, let's create one:

```py
resnet50d = ResnetModelForImageClassification(resnet50d_config)
```

Again, you can use any of the methods of [`PreTrainedModel`], like [`~PreTrainedModel.save_pretrained`] or
[`~PreTrainedModel.push_to_hub`].


key: _node_content
string_value:
{"id_": "fd3e485b-6d00-4543-b4b3-737434b34037", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e6d92be0-7075-45f5-9f81-d8575684661e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "d4bfffcd0f7b9b6df805a1badab8bc6b6283a03c090d79361066e6a1f265c843", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d9b355ab-151b-48a2-b912-6b6a8c98bdd5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "68608649c8f6a1ab818229bd176b0613c06fd854e4bdbad3f5e7530a6380883a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ab919d34-75e0-4bee-bdd9-ad8946dfd79b", "node_type": "1", "metadata": {}, "hash": "117424f2eb376189b1643b08affd0393634810d9fbe0256ba1afcf05b54ba6d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7392, "end_char_idx": 10749, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The line that sets the `config_class` is not mandatory, unless
you want to register your model with the auto classes (see last section).

<Tip>

If your model is very similar to a model inside the library, you can re-use the same configuration as this model.

</Tip>

You can have your model return anything you want, but returning a dictionary like we did for
`ResnetModelForImageClassification`, with the loss included when labels are passed, will make your model directly
usable inside the [`Trainer`] class. Using another output format is fine as long as you are planning on using your own
training loop or another library for training.

Now that we have our model class, let's create one:

```py
resnet50d = ResnetModelForImageClassification(resnet50d_config)
```

Again, you can use any of the methods of [`PreTrainedModel`], like [`~PreTrainedModel.save_pretrained`] or
[`~PreTrainedModel.push_to_hub`]. We will use the second in the next section, and see how to push the model weights
with the code of our model. But first, let's load some pretrained weights inside our model.

In your own use case, you will probably be training your custom model on your own data. To go fast for this tutorial,
we will use the pretrained version of the resnet50d. Since our model is just a wrapper around it, it's going to be
easy to transfer those weights:

```py
import timm

pretrained_model = timm.create_model("resnet50d", pretrained=True)
resnet50d.model.load_state_dict(pretrained_model.state_dict())
```

Now let's see how to make sure that when we do [`~PreTrainedModel.save_pretrained`] or [`~PreTrainedModel.push_to_hub`], the
code of the model is saved.

## Registering a model with custom code to the auto classes

If you are writing a library that extends 🤗 Transformers, you may want to extend the auto classes to include your own
model. This is different from pushing the code to the Hub in the sense that users will need to import your library to
get the custom models (contrarily to automatically downloading the model code from the Hub).

As long as your config has a `model_type` attribute that is different from existing model types, and that your model
classes have the right `config_class` attributes, you can just add them to the auto classes like this:

```py
from transformers import AutoConfig, AutoModel, AutoModelForImageClassification

AutoConfig.register("resnet", ResnetConfig)
AutoModel.register(ResnetConfig, ResnetModel)
AutoModelForImageClassification.register(ResnetConfig, ResnetModelForImageClassification)
```

Note that the first argument used when registering your custom config to [`AutoConfig`] needs to match the `model_type`
of your custom config, and the first argument used when registering your custom models to any auto model class needs
to match the `config_class` of those models.

## Sending the code to the Hub

<Tip warning={true}>

This API is experimental and may have some slight breaking changes in the next releases.

</Tip>

First, make sure your model is fully defined in a `.py` file. It can rely on relative imports to some other files as
long as all the files are in the same directory (we don't support submodules for this feature yet). For our example,
we'll define a `modeling_resnet.py` file and a `configuration_resnet.py` file in a folder of the current working
directory named `resnet_model`.


key: _node_content
string_value:
{"id_": "ab919d34-75e0-4bee-bdd9-ad8946dfd79b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e6d92be0-7075-45f5-9f81-d8575684661e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "d4bfffcd0f7b9b6df805a1badab8bc6b6283a03c090d79361066e6a1f265c843", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fd3e485b-6d00-4543-b4b3-737434b34037", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "99ef911017cc6e68e8fa0c4be43a05341588aab193b7c0cb918ec2ca36125183", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e8028336-f61f-43e6-8d3d-5d44628ba4b5", "node_type": "1", "metadata": {}, "hash": "ed2c028b3ce8b65cf66d88bd536afe7300e259122596712e4ab91861ccecb80b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9052, "end_char_idx": 12476, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Registering a model with custom code to the auto classes

If you are writing a library that extends 🤗 Transformers, you may want to extend the auto classes to include your own
model. This is different from pushing the code to the Hub in the sense that users will need to import your library to
get the custom models (contrarily to automatically downloading the model code from the Hub).

As long as your config has a `model_type` attribute that is different from existing model types, and that your model
classes have the right `config_class` attributes, you can just add them to the auto classes like this:

```py
from transformers import AutoConfig, AutoModel, AutoModelForImageClassification

AutoConfig.register("resnet", ResnetConfig)
AutoModel.register(ResnetConfig, ResnetModel)
AutoModelForImageClassification.register(ResnetConfig, ResnetModelForImageClassification)
```

Note that the first argument used when registering your custom config to [`AutoConfig`] needs to match the `model_type`
of your custom config, and the first argument used when registering your custom models to any auto model class needs
to match the `config_class` of those models.

## Sending the code to the Hub

<Tip warning={true}>

This API is experimental and may have some slight breaking changes in the next releases.

</Tip>

First, make sure your model is fully defined in a `.py` file. It can rely on relative imports to some other files as
long as all the files are in the same directory (we don't support submodules for this feature yet). For our example,
we'll define a `modeling_resnet.py` file and a `configuration_resnet.py` file in a folder of the current working
directory named `resnet_model`. The configuration file contains the code for `ResnetConfig` and the modeling file
contains the code of `ResnetModel` and `ResnetModelForImageClassification`.

```
.
└── resnet_model
    ├── __init__.py
    ├── configuration_resnet.py
    └── modeling_resnet.py
```

The `__init__.py` can be empty, it's just there so that Python detects `resnet_model` can be use as a module.

<Tip warning={true}>

If copying a modeling files from the library, you will need to replace all the relative imports at the top of the file
to import from the `transformers` package.

</Tip>

Note that you can re-use (or subclass) an existing configuration/model.

To share your model with the community, follow those steps: first import the ResNet model and config from the newly
created files:

```py
from resnet_model.configuration_resnet import ResnetConfig
from resnet_model.modeling_resnet import ResnetModel, ResnetModelForImageClassification
```

Then you have to tell the library you want to copy the code files of those objects when using the `save_pretrained`
method and properly register them with a given Auto class (especially for models), just run:

```py
ResnetConfig.register_for_auto_class()
ResnetModel.register_for_auto_class("AutoModel")
ResnetModelForImageClassification.register_for_auto_class("AutoModelForImageClassification")
```

Note that there is no need to specify an auto class for the configuration (there is only one auto class for them,
[`AutoConfig`]) but it's different for models. Your custom model could be suitable for many different tasks, so you
have to specify which one of the auto classes is the correct one for your model.

<Tip>

Use `register_for_auto_class()` if you want the code files to be copied.


key: _node_content
string_value:
{"id_": "e8028336-f61f-43e6-8d3d-5d44628ba4b5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e6d92be0-7075-45f5-9f81-d8575684661e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "d4bfffcd0f7b9b6df805a1badab8bc6b6283a03c090d79361066e6a1f265c843", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ab919d34-75e0-4bee-bdd9-ad8946dfd79b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "4debc4eab8b3dcf3e64d67e98588b003ddf3c02aee247bd4618eb264b4aef55c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed6c9b16-072b-416d-a62e-b37c68f0fd54", "node_type": "1", "metadata": {}, "hash": "0762a7b506e827b292dbf7086cd1992fd1d3acba3c252f22cbcd0b429e11cb15", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10750, "end_char_idx": 13933, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The configuration file contains the code for `ResnetConfig` and the modeling file
contains the code of `ResnetModel` and `ResnetModelForImageClassification`.

```
.
└── resnet_model
    ├── __init__.py
    ├── configuration_resnet.py
    └── modeling_resnet.py
```

The `__init__.py` can be empty, it's just there so that Python detects `resnet_model` can be use as a module.

<Tip warning={true}>

If copying a modeling files from the library, you will need to replace all the relative imports at the top of the file
to import from the `transformers` package.

</Tip>

Note that you can re-use (or subclass) an existing configuration/model.

To share your model with the community, follow those steps: first import the ResNet model and config from the newly
created files:

```py
from resnet_model.configuration_resnet import ResnetConfig
from resnet_model.modeling_resnet import ResnetModel, ResnetModelForImageClassification
```

Then you have to tell the library you want to copy the code files of those objects when using the `save_pretrained`
method and properly register them with a given Auto class (especially for models), just run:

```py
ResnetConfig.register_for_auto_class()
ResnetModel.register_for_auto_class("AutoModel")
ResnetModelForImageClassification.register_for_auto_class("AutoModelForImageClassification")
```

Note that there is no need to specify an auto class for the configuration (there is only one auto class for them,
[`AutoConfig`]) but it's different for models. Your custom model could be suitable for many different tasks, so you
have to specify which one of the auto classes is the correct one for your model.

<Tip>

Use `register_for_auto_class()` if you want the code files to be copied. If you instead prefer to use code on the Hub from another repo, 
you don't need to call it. In cases where there's more than one auto class, you can modify the `config.json` directly using the 
following structure:

```json
"auto_map": {     
	"AutoConfig": "<your-repo-name>--<config-name>",     
	"AutoModel": "<your-repo-name>--<config-name>",
	"AutoModelFor<Task>": "<your-repo-name>--<config-name>",    
},
```

</Tip>

Next, let's create the config and models as we did before:

```py
resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d = ResnetModelForImageClassification(resnet50d_config)

pretrained_model = timm.create_model("resnet50d", pretrained=True)
resnet50d.model.load_state_dict(pretrained_model.state_dict())
```

Now to send the model to the Hub, make sure you are logged in. Either run in your terminal:

```bash
huggingface-cli login
```

or from a notebook:

```py
from huggingface_hub import notebook_login

notebook_login()
```

You can then push to your own namespace (or an organization you are a member of) like this:

```py
resnet50d.push_to_hub("custom-resnet50d")
```

On top of the modeling weights and the configuration in json format, this also copied the modeling and
configuration `.py` files in the folder `custom-resnet50d` and uploaded the result to the Hub. You can check the result
in this [model repo](https://huggingface.co/sgugger/custom-resnet50d).


key: _node_content
string_value:
{"id_": "ed6c9b16-072b-416d-a62e-b37c68f0fd54", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e6d92be0-7075-45f5-9f81-d8575684661e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "d4bfffcd0f7b9b6df805a1badab8bc6b6283a03c090d79361066e6a1f265c843", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8028336-f61f-43e6-8d3d-5d44628ba4b5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models", "title": "Building custom models", "tokens": 3681, "retrieve_doc": true, "source": "transformers"}, "hash": "bf3efe4a60a0a74dcb212eff100e969071f5dda5df8e4e3033dc37cbcf3f4db0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12477, "end_char_idx": 15296, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you instead prefer to use code on the Hub from another repo, 
you don't need to call it. In cases where there's more than one auto class, you can modify the `config.json` directly using the 
following structure:

```json
"auto_map": {     
	"AutoConfig": "<your-repo-name>--<config-name>",     
	"AutoModel": "<your-repo-name>--<config-name>",
	"AutoModelFor<Task>": "<your-repo-name>--<config-name>",    
},
```

</Tip>

Next, let's create the config and models as we did before:

```py
resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d = ResnetModelForImageClassification(resnet50d_config)

pretrained_model = timm.create_model("resnet50d", pretrained=True)
resnet50d.model.load_state_dict(pretrained_model.state_dict())
```

Now to send the model to the Hub, make sure you are logged in. Either run in your terminal:

```bash
huggingface-cli login
```

or from a notebook:

```py
from huggingface_hub import notebook_login

notebook_login()
```

You can then push to your own namespace (or an organization you are a member of) like this:

```py
resnet50d.push_to_hub("custom-resnet50d")
```

On top of the modeling weights and the configuration in json format, this also copied the modeling and
configuration `.py` files in the folder `custom-resnet50d` and uploaded the result to the Hub. You can check the result
in this [model repo](https://huggingface.co/sgugger/custom-resnet50d).

See the [sharing tutorial](model_sharing) for more information on the push to Hub method.

## Using a model with custom code

You can use any configuration, model or tokenizer with custom code files in its repository with the auto-classes and
the `from_pretrained` method. All files and code uploaded to the Hub are scanned for malware (refer to the [Hub security](https://huggingface.co/docs/hub/security#malware-scanning) documentation for more information), but you should still 
review the model code and author to avoid executing malicious code on your machine. Set `trust_remote_code=True` to use
a model with custom code:

```py
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained("sgugger/custom-resnet50d", trust_remote_code=True)
```

It is also strongly encouraged to pass a commit hash as a `revision` to make sure the author of the models did not
update the code with some malicious new lines (unless you fully trust the authors of the models).

```py
commit_hash = "ed94a7c6247d8aedce4647f00f20de6875b5b292"
model = AutoModelForImageClassification.from_pretrained(
    "sgugger/custom-resnet50d", trust_remote_code=True, revision=commit_hash
)
```

Note that when browsing the commit history of the model repo on the Hub, there is a button to easily copy the commit
hash of any commit.


key: _node_content
string_value:
{"id_": "f499d41e-31df-439f-8447-462129426207", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0b7bc08a-4664-4c75-a8be-6db77543a36a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "hash": "37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cb0d9690-6530-4254-bf3a-61f20d312294", "node_type": "1", "metadata": {}, "hash": "eee08c127d0da344197c7f0d0d399a0bf975872c44e75bc1bb990ea8f7485437", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2752, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Load pretrained instances with an AutoClass

With so many different Transformer architectures, it can be challenging to create one for your checkpoint. As a part of 🤗 Transformers core philosophy to make the library easy, simple and flexible to use, an `AutoClass` automatically infers and loads the correct architecture from a given checkpoint. The `from_pretrained()` method lets you quickly load a pretrained model for any architecture so you don't have to devote time and resources to train a model from scratch. Producing this type of checkpoint-agnostic code means if your code works for one checkpoint, it will work with another checkpoint - as long as it was trained for a similar task - even if the architecture is different.

<Tip>

Remember, architecture refers to the skeleton of the model and checkpoints are the weights for a given architecture. For example, [BERT](https://huggingface.co/google-bert/bert-base-uncased) is an architecture, while `google-bert/bert-base-uncased` is a checkpoint. Model is a general term that can mean either architecture or checkpoint.

</Tip>

In this tutorial, learn to:

* Load a pretrained tokenizer.
* Load a pretrained image processor
* Load a pretrained feature extractor.
* Load a pretrained processor.
* Load a pretrained model.
* Load a model as a backbone.

## AutoTokenizer

Nearly every NLP task begins with a tokenizer. A tokenizer converts your input into a format that can be processed by the model.

Load a tokenizer with [`AutoTokenizer.from_pretrained`]:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
```

Then tokenize your input as shown below:

```py
>>> sequence = "In a hole in the ground there lived a hobbit."
>>> print(tokenizer(sequence))
{'input_ids': [101, 1999, 1037, 4920, 1999, 1996, 2598, 2045, 2973, 1037, 7570, 10322, 4183, 1012, 102], 
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

## AutoImageProcessor

For vision tasks, an image processor processes the image into the correct input format.

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
```

## AutoBackbone

<div style="text-align: center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stages.png">
    <figcaption class="mt-2 text-center text-sm text-gray-500">A Swin backbone with multiple stages for outputting a feature map.</figcaption>
</div>

The [`AutoBackbone`] lets you use pretrained models as backbones to get feature maps from different stages of the backbone.


key: _node_content
string_value:
{"id_": "cb0d9690-6530-4254-bf3a-61f20d312294", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0b7bc08a-4664-4c75-a8be-6db77543a36a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "hash": "37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f499d41e-31df-439f-8447-462129426207", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "hash": "e9bdbb56903aa09378be5712c61e971e26290f1b48b21abe72eb8ab3d18d8c75", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "558e0aab-dfd9-41f6-a109-37eded242fff", "node_type": "1", "metadata": {}, "hash": "28f80ca76a2be400d18f73413702348ada1cccdb895d9037527582d29e9b9a5e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1768, "end_char_idx": 4537, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> print(tokenizer(sequence))
{'input_ids': [101, 1999, 1037, 4920, 1999, 1996, 2598, 2045, 2973, 1037, 7570, 10322, 4183, 1012, 102], 
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

## AutoImageProcessor

For vision tasks, an image processor processes the image into the correct input format.

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
```

## AutoBackbone

<div style="text-align: center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stages.png">
    <figcaption class="mt-2 text-center text-sm text-gray-500">A Swin backbone with multiple stages for outputting a feature map.</figcaption>
</div>

The [`AutoBackbone`] lets you use pretrained models as backbones to get feature maps from different stages of the backbone. You should specify one of the following parameters in [`~PretrainedConfig.from_pretrained`]:

* `out_indices` is the index of the layer you'd like to get the feature map from
* `out_features` is the name of the layer you'd like to get the feature map from

These parameters can be used interchangeably, but if you use both, make sure they're aligned with each other! If you don't pass any of these parameters, the backbone returns the feature map from the last layer.

<div style="text-align: center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stage%201.png">
    <figcaption class="mt-2 text-center text-sm text-gray-500">A feature map from the first stage of the backbone. The patch partition refers to the model stem.</figcaption>
</div>

For example, in the above diagram, to return the feature map from the first stage of the Swin backbone, you can set `out_indices=(1,)`:

```py
>>> from transformers import AutoImageProcessor, AutoBackbone
>>> import torch
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> processor = AutoImageProcessor.from_pretrained("microsoft/swin-tiny-patch4-window7-224")
>>> model = AutoBackbone.from_pretrained("microsoft/swin-tiny-patch4-window7-224", out_indices=(1,))

>>> inputs = processor(image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> feature_maps = outputs.feature_maps
```

Now you can access the `feature_maps` object from the first stage of the backbone:

```py
>>> list(feature_maps[0].shape)
[1, 96, 56, 56]
```

## AutoFeatureExtractor

For audio tasks, a feature extractor processes the audio signal the correct input format.


key: _node_content
string_value:
{"id_": "558e0aab-dfd9-41f6-a109-37eded242fff", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0b7bc08a-4664-4c75-a8be-6db77543a36a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "hash": "37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cb0d9690-6530-4254-bf3a-61f20d312294", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "hash": "cb8aae3c4142340de9f8b129748b3cf895d69434a90da83d6851c9ff1b6c9356", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5761229-6977-4257-9d8b-832bf3f7c1ba", "node_type": "1", "metadata": {}, "hash": "381199319bccdd20b339d58e6c1dce48138341d80a3e5d9655cf689f789194ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3120, "end_char_idx": 6276, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you don't pass any of these parameters, the backbone returns the feature map from the last layer.

<div style="text-align: center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stage%201.png">
    <figcaption class="mt-2 text-center text-sm text-gray-500">A feature map from the first stage of the backbone. The patch partition refers to the model stem.</figcaption>
</div>

For example, in the above diagram, to return the feature map from the first stage of the Swin backbone, you can set `out_indices=(1,)`:

```py
>>> from transformers import AutoImageProcessor, AutoBackbone
>>> import torch
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> processor = AutoImageProcessor.from_pretrained("microsoft/swin-tiny-patch4-window7-224")
>>> model = AutoBackbone.from_pretrained("microsoft/swin-tiny-patch4-window7-224", out_indices=(1,))

>>> inputs = processor(image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> feature_maps = outputs.feature_maps
```

Now you can access the `feature_maps` object from the first stage of the backbone:

```py
>>> list(feature_maps[0].shape)
[1, 96, 56, 56]
```

## AutoFeatureExtractor

For audio tasks, a feature extractor processes the audio signal the correct input format.

Load a feature extractor with [`AutoFeatureExtractor.from_pretrained`]:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained(
...     "ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition"
... )
```

## AutoProcessor

Multimodal tasks require a processor that combines two types of preprocessing tools. For example, the [LayoutLMV2](model_doc/layoutlmv2) model requires an image processor to handle images and a tokenizer to handle text; a processor combines both of them.

Load a processor with [`AutoProcessor.from_pretrained`]:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")
```

## AutoModel

<frameworkcontent>
<pt>
The `AutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks). For example, load a model for sequence classification with [`AutoModelForSequenceClassification.from_pretrained`]:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse the same checkpoint to load an architecture for a different task:

```py
>>> from transformers import AutoModelForTokenClassification

>>> model = AutoModelForTokenClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

<Tip warning={true}>

For PyTorch models, the `from_pretrained()` method uses `torch.load()` which internally uses `pickle` and is known to be insecure. In general, never load a model that could have come from an untrusted source, or that could have been tampered with.


key: _node_content
string_value:
{"id_": "e5761229-6977-4257-9d8b-832bf3f7c1ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0b7bc08a-4664-4c75-a8be-6db77543a36a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "hash": "37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "558e0aab-dfd9-41f6-a109-37eded242fff", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "hash": "23e56357fb82cc16e3c1f923734aa62b333ecb1fc9ccb2e68062bb5e9068c8d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5e72d30-85fa-483a-b3fb-1134cd93dd45", "node_type": "1", "metadata": {}, "hash": "54ab0752e326d3f02721787158ef44640b3c6cf1eb3e6d64ded4177f27aeb082", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4539, "end_char_idx": 7409, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Load a feature extractor with [`AutoFeatureExtractor.from_pretrained`]:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained(
...     "ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition"
... )
```

## AutoProcessor

Multimodal tasks require a processor that combines two types of preprocessing tools. For example, the [LayoutLMV2](model_doc/layoutlmv2) model requires an image processor to handle images and a tokenizer to handle text; a processor combines both of them.

Load a processor with [`AutoProcessor.from_pretrained`]:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")
```

## AutoModel

<frameworkcontent>
<pt>
The `AutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks). For example, load a model for sequence classification with [`AutoModelForSequenceClassification.from_pretrained`]:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse the same checkpoint to load an architecture for a different task:

```py
>>> from transformers import AutoModelForTokenClassification

>>> model = AutoModelForTokenClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

<Tip warning={true}>

For PyTorch models, the `from_pretrained()` method uses `torch.load()` which internally uses `pickle` and is known to be insecure. In general, never load a model that could have come from an untrusted source, or that could have been tampered with. This security risk is partially mitigated for public models hosted on the Hugging Face Hub, which are [scanned for malware](https://huggingface.co/docs/hub/security-malware) at each commit. See the [Hub documentation](https://huggingface.co/docs/hub/security) for best practices like [signed commit verification](https://huggingface.co/docs/hub/security-gpg#signing-commits-with-gpg) with GPG.

TensorFlow and Flax checkpoints are not affected, and can be loaded within PyTorch architectures using the `from_tf` and `from_flax` kwargs for the `from_pretrained` method to circumvent this issue.

</Tip>

Generally, we recommend using the `AutoTokenizer` class and the `AutoModelFor` class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next [tutorial](preprocessing), learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.
</pt>
<tf>
Finally, the `TFAutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks).


key: _node_content
string_value:
{"id_": "d5e72d30-85fa-483a-b3fb-1134cd93dd45", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0b7bc08a-4664-4c75-a8be-6db77543a36a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "hash": "37c9b5eded3d76dec9c0881b4655629180885e5d87a64c19dc0f05978a5668d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5761229-6977-4257-9d8b-832bf3f7c1ba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "transformers"}, "hash": "073262c53e3caf9f7ca96d4e9eeac35c16768dbbaae8145135ccb6ba2f865c2f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6160, "end_char_idx": 8354, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In general, never load a model that could have come from an untrusted source, or that could have been tampered with. This security risk is partially mitigated for public models hosted on the Hugging Face Hub, which are [scanned for malware](https://huggingface.co/docs/hub/security-malware) at each commit. See the [Hub documentation](https://huggingface.co/docs/hub/security) for best practices like [signed commit verification](https://huggingface.co/docs/hub/security-gpg#signing-commits-with-gpg) with GPG.

TensorFlow and Flax checkpoints are not affected, and can be loaded within PyTorch architectures using the `from_tf` and `from_flax` kwargs for the `from_pretrained` method to circumvent this issue.

</Tip>

Generally, we recommend using the `AutoTokenizer` class and the `AutoModelFor` class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next [tutorial](preprocessing), learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.
</pt>
<tf>
Finally, the `TFAutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks). For example, load a model for sequence classification with [`TFAutoModelForSequenceClassification.from_pretrained`]:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse the same checkpoint to load an architecture for a different task:

```py
>>> from transformers import TFAutoModelForTokenClassification

>>> model = TFAutoModelForTokenClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Generally, we recommend using the `AutoTokenizer` class and the `TFAutoModelFor` class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next [tutorial](preprocessing), learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "b0f7b7ff-d37d-48eb-ae44-8c283529ca47", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a2753f6-6769-4b3e-9289-96833ffad726", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "hash": "e3f8e93acb5d04a644376f51295677234286b733dcf7411faf693460ef9a0458", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "30e9d6f4-c108-46a1-949a-52241cb8dbda", "node_type": "1", "metadata": {}, "hash": "4e29a365cf288d80c7dab262266ef79d8dbebbd54f2eec6e1ee4c9927ab70a31", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3101, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CPU inference

With some optimizations, it is possible to efficiently run large model inference on a CPU. One of these optimization techniques involves compiling the PyTorch code into an intermediate format for high-performance environments like C++. The other technique fuses multiple operations into one kernel to reduce the overhead of running each operation separately.

You'll learn how to use [BetterTransformer](https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/) for faster inference, and how to convert your PyTorch code to [TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html). If you're using an Intel CPU, you can also use [graph optimizations](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features.html#graph-optimization) from [Intel Extension for PyTorch](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/index.html) to boost inference speed even more. Finally, learn how to use 🤗 Optimum to accelerate inference with ONNX Runtime or OpenVINO (if you're using an Intel CPU).

## BetterTransformer

BetterTransformer accelerates inference with its fastpath (native PyTorch specialized implementation of Transformer functions) execution. The two optimizations in the fastpath execution are:

1. fusion, which combines multiple sequential operations into a single "kernel" to reduce the number of computation steps
2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors

BetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention).

<Tip>

BetterTransformer is not supported for all models. Check this [list](https://huggingface.co/docs/optimum/bettertransformer/overview#supported-models) to see if a model supports BetterTransformer.

</Tip>

Before you start, make sure you have 🤗 Optimum [installed](https://huggingface.co/docs/optimum/installation).

Enable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigcode/starcoder")
model.to_bettertransformer()
```

## TorchScript

TorchScript is an intermediate PyTorch model representation that can be run in production environments where performance is important. You can train a model in PyTorch and then export it to TorchScript to free the model from Python performance constraints. PyTorch [traces](https://pytorch.org/docs/stable/generated/torch.jit.trace.html) a model to return a [`ScriptFunction`] that is optimized with just-in-time compilation (JIT). Compared to the default eager mode, JIT mode in PyTorch typically yields better performance for inference using optimization techniques like operator fusion.

For a gentle introduction to TorchScript, see the [Introduction to PyTorch TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) tutorial.


key: _node_content
string_value:
{"id_": "30e9d6f4-c108-46a1-949a-52241cb8dbda", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a2753f6-6769-4b3e-9289-96833ffad726", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "hash": "e3f8e93acb5d04a644376f51295677234286b733dcf7411faf693460ef9a0458", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b0f7b7ff-d37d-48eb-ae44-8c283529ca47", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "hash": "6886338a93a580319ed09db781e0185c0ea938e76339698d6f97fec66ddc4ae9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c230de7f-3bf1-40b6-96b8-f4e4de24ebe2", "node_type": "1", "metadata": {}, "hash": "17a3c268249194838a77a1e6aaf373a827a44c6d8a8bab35da0464c6bd50c7a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1265, "end_char_idx": 4457, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The two optimizations in the fastpath execution are:

1. fusion, which combines multiple sequential operations into a single "kernel" to reduce the number of computation steps
2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors

BetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention).

<Tip>

BetterTransformer is not supported for all models. Check this [list](https://huggingface.co/docs/optimum/bettertransformer/overview#supported-models) to see if a model supports BetterTransformer.

</Tip>

Before you start, make sure you have 🤗 Optimum [installed](https://huggingface.co/docs/optimum/installation).

Enable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigcode/starcoder")
model.to_bettertransformer()
```

## TorchScript

TorchScript is an intermediate PyTorch model representation that can be run in production environments where performance is important. You can train a model in PyTorch and then export it to TorchScript to free the model from Python performance constraints. PyTorch [traces](https://pytorch.org/docs/stable/generated/torch.jit.trace.html) a model to return a [`ScriptFunction`] that is optimized with just-in-time compilation (JIT). Compared to the default eager mode, JIT mode in PyTorch typically yields better performance for inference using optimization techniques like operator fusion.

For a gentle introduction to TorchScript, see the [Introduction to PyTorch TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) tutorial.

With the [`Trainer`] class, you can enable JIT mode for CPU inference by setting the `--jit_mode_eval` flag:

```bash
python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--jit_mode_eval
```

<Tip warning={true}>

For PyTorch >= 1.14.0, JIT-mode could benefit any model for prediction and evaluation since the dict input is supported in `jit.trace`.

For PyTorch < 1.14.0, JIT-mode could benefit a model if its forward parameter order matches the tuple input order in `jit.trace`, such as a question-answering model. If the forward parameter order does not match the tuple input order in `jit.trace`, like a text classification model, `jit.trace` will fail and we are capturing this with the exception here to make it fallback. Logging is used to notify users.

</Tip>

## IPEX graph optimization

Intel® Extension for PyTorch (IPEX) provides further optimizations in JIT mode for Intel CPUs, and we recommend combining it with TorchScript for even faster performance. The IPEX [graph optimization](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/graph_optimization.html) fuses operations like Multi-head attention, Concat Linear, Linear + Add, Linear + Gelu, Add + LayerNorm, and more.


key: _node_content
string_value:
{"id_": "c230de7f-3bf1-40b6-96b8-f4e4de24ebe2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a2753f6-6769-4b3e-9289-96833ffad726", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "hash": "e3f8e93acb5d04a644376f51295677234286b733dcf7411faf693460ef9a0458", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "30e9d6f4-c108-46a1-949a-52241cb8dbda", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "hash": "529873a962bd4b42fbaa1abb350f17ec150d62ba64966b0a98bbd155cd850fc0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4c77aeb0-8ed3-4151-8371-465c890dafff", "node_type": "1", "metadata": {}, "hash": "264dbe468f53d819287f59c7b8637853f250e1a7d7872a587eb70aa3d07a9945", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2929, "end_char_idx": 5674, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For a gentle introduction to TorchScript, see the [Introduction to PyTorch TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) tutorial.

With the [`Trainer`] class, you can enable JIT mode for CPU inference by setting the `--jit_mode_eval` flag:

```bash
python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--jit_mode_eval
```

<Tip warning={true}>

For PyTorch >= 1.14.0, JIT-mode could benefit any model for prediction and evaluation since the dict input is supported in `jit.trace`.

For PyTorch < 1.14.0, JIT-mode could benefit a model if its forward parameter order matches the tuple input order in `jit.trace`, such as a question-answering model. If the forward parameter order does not match the tuple input order in `jit.trace`, like a text classification model, `jit.trace` will fail and we are capturing this with the exception here to make it fallback. Logging is used to notify users.

</Tip>

## IPEX graph optimization

Intel® Extension for PyTorch (IPEX) provides further optimizations in JIT mode for Intel CPUs, and we recommend combining it with TorchScript for even faster performance. The IPEX [graph optimization](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/graph_optimization.html) fuses operations like Multi-head attention, Concat Linear, Linear + Add, Linear + Gelu, Add + LayerNorm, and more.

To take advantage of these graph optimizations, make sure you have IPEX [installed](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/installation.html):

```bash
pip install intel_extension_for_pytorch
```

Set the `--use_ipex` and `--jit_mode_eval` flags in the [`Trainer`] class to enable JIT mode with the graph optimizations:

```bash
python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--use_ipex \
--jit_mode_eval
```

## 🤗 Optimum

<Tip>

Learn more details about using ORT with 🤗 Optimum in the [Optimum Inference with ONNX Runtime](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models) guide. This section only provides a brief and simple example.

</Tip>

ONNX Runtime (ORT) is a model accelerator that runs inference on CPUs by default. ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers, without making too many changes to your code. You only need to replace the 🤗 Transformers `AutoClass` with its equivalent [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and load a checkpoint in the ONNX format.


key: _node_content
string_value:
{"id_": "4c77aeb0-8ed3-4151-8371-465c890dafff", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a2753f6-6769-4b3e-9289-96833ffad726", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "hash": "e3f8e93acb5d04a644376f51295677234286b733dcf7411faf693460ef9a0458", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c230de7f-3bf1-40b6-96b8-f4e4de24ebe2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "transformers"}, "hash": "6c7809a940cbcb0efb7a71915d45077fbec8544a76b7d831ad63c63aa5dc4c0a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4206, "end_char_idx": 6711, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The IPEX [graph optimization](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/graph_optimization.html) fuses operations like Multi-head attention, Concat Linear, Linear + Add, Linear + Gelu, Add + LayerNorm, and more.

To take advantage of these graph optimizations, make sure you have IPEX [installed](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/installation.html):

```bash
pip install intel_extension_for_pytorch
```

Set the `--use_ipex` and `--jit_mode_eval` flags in the [`Trainer`] class to enable JIT mode with the graph optimizations:

```bash
python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--use_ipex \
--jit_mode_eval
```

## 🤗 Optimum

<Tip>

Learn more details about using ORT with 🤗 Optimum in the [Optimum Inference with ONNX Runtime](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models) guide. This section only provides a brief and simple example.

</Tip>

ONNX Runtime (ORT) is a model accelerator that runs inference on CPUs by default. ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers, without making too many changes to your code. You only need to replace the 🤗 Transformers `AutoClass` with its equivalent [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and load a checkpoint in the ONNX format.

For example, if you're running inference on a question answering task, load the [optimum/roberta-base-squad2](https://huggingface.co/optimum/roberta-base-squad2) checkpoint which contains a `model.onnx` file:

```py
from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForQuestionAnswering

model = ORTModelForQuestionAnswering.from_pretrained("optimum/roberta-base-squad2")
tokenizer = AutoTokenizer.from_pretrained("deepset/roberta-base-squad2")

onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer)

question = "What's my name?"
context = "My name is Philipp and I live in Nuremberg."
pred = onnx_qa(question, context)
```

If you have an Intel CPU, take a look at 🤗 [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) which supports a variety of compression techniques (quantization, pruning, knowledge distillation) and tools for converting models to the [OpenVINO](https://huggingface.co/docs/optimum/intel/inference) format for higher performance inference.


key: _node_content
string_value:
{"id_": "81f45c62-f4c3-4ca2-a422-99fe1cc33d20", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/bertology", "title": "BERTology", "tokens": 562, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d198add-47b8-4174-b5cb-28d1379f70a4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/bertology", "title": "BERTology", "tokens": 562, "retrieve_doc": true, "source": "transformers"}, "hash": "10e5a6a2247446e6640196ae886427aacd1f744b4ef227463338fd0cb719316f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1576, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BERTology

There is a growing field of study concerned with investigating the inner working of large-scale transformers like BERT
(that some call "BERTology"). Some good examples of this field are:


- BERT Rediscovers the Classical NLP Pipeline by Ian Tenney, Dipanjan Das, Ellie Pavlick:
  https://arxiv.org/abs/1905.05950
- Are Sixteen Heads Really Better than One? by Paul Michel, Omer Levy, Graham Neubig: https://arxiv.org/abs/1905.10650
- What Does BERT Look At? An Analysis of BERT's Attention by Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D.
  Manning: https://arxiv.org/abs/1906.04341
- CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure: https://arxiv.org/abs/2210.04633

In order to help this new field develop, we have included a few additional features in the BERT/GPT/GPT-2 models to
help people access the inner representations, mainly adapted from the great work of Paul Michel
(https://arxiv.org/abs/1905.10650):


- accessing all the hidden-states of BERT/GPT/GPT-2,
- accessing all the attention weights for each head of BERT/GPT/GPT-2,
- retrieving heads output values and gradients to be able to compute head importance score and prune head as explained
  in https://arxiv.org/abs/1905.10650.

To help you understand and use these features, we have added a specific example script: [bertology.py](https://github.com/huggingface/transformers/tree/main/examples/research_projects/bertology/run_bertology.py) while extract information and prune a model pre-trained on
GLUE.


key: _node_content
string_value:
{"id_": "3e1627ef-adbb-476c-b47f-985675aebd05", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9daadbd-294f-4c73-bb06-bbfaf394994a", "node_type": "1", "metadata": {}, "hash": "d2699a1a364d3677a2084a1a0bde7a9e246ccda461215d92ed5ac6ef92bb8a82", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2378, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Testing


Let's take a look at how 🤗 Transformers models are tested and how you can write new tests and improve the existing ones.

There are 2 test suites in the repository:

1. `tests` -- tests for the general API
2. `examples` -- tests primarily for various applications that aren't part of the API

## How transformers are tested

1. Once a PR is submitted it gets tested with 9 CircleCi jobs. Every new commit to that PR gets retested. These jobs
   are defined in this [config file](https://github.com/huggingface/transformers/tree/main/.circleci/config.yml), so that if needed you can reproduce the same
   environment on your machine.

   These CI jobs don't run `@slow` tests.

2. There are 3 jobs run by [github actions](https://github.com/huggingface/transformers/actions):

   - [torch hub integration](https://github.com/huggingface/transformers/tree/main/.github/workflows/github-torch-hub.yml): checks whether torch hub
     integration works.

   - [self-hosted (push)](https://github.com/huggingface/transformers/tree/main/.github/workflows/self-push.yml): runs fast tests on GPU only on commits on
     `main`. It only runs if a commit on `main` has updated the code in one of the following folders: `src`,
     `tests`, `.github` (to prevent running on added model cards, notebooks, etc.)

   - [self-hosted runner](https://github.com/huggingface/transformers/tree/main/.github/workflows/self-scheduled.yml): runs normal and slow tests on GPU in
     `tests` and `examples`:

```bash
RUN_SLOW=1 pytest tests/
RUN_SLOW=1 pytest examples/
```

   The results can be observed [here](https://github.com/huggingface/transformers/actions).



## Running tests



### Choosing which tests to run

This document goes into many details of how tests can be run. If after reading everything, you need even more details
you will find them [here](https://docs.pytest.org/en/latest/usage.html).

Here are some most useful ways of running tests.

Run all:

```console
pytest
```

or:

```bash
make test
```

Note that the latter is defined as:

```bash
python -m pytest -n auto --dist=loadfile -s -v ./tests/
```

which tells pytest to:

- run as many test processes as they are CPU cores (which could be too many if you don't have a ton of RAM!)
- ensure that all tests from the same file will be run by the same test process
- do not capture output
- run in verbose mode


key: _node_content
string_value:
{"id_": "a9daadbd-294f-4c73-bb06-bbfaf394994a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3e1627ef-adbb-476c-b47f-985675aebd05", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "0f3d4ffc17eb3df9c3e2a299c2ce0da911cd1049f99a835c24182b1ea7c5697a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b25a3c49-43c8-4489-8f59-05275ebc3fff", "node_type": "1", "metadata": {}, "hash": "40eebf1d49ea21e96b1b5ccd5ab9f8f4ff34ffc545604b923e603f69d4c8b88c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1658, "end_char_idx": 4733, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Running tests



### Choosing which tests to run

This document goes into many details of how tests can be run. If after reading everything, you need even more details
you will find them [here](https://docs.pytest.org/en/latest/usage.html).

Here are some most useful ways of running tests.

Run all:

```console
pytest
```

or:

```bash
make test
```

Note that the latter is defined as:

```bash
python -m pytest -n auto --dist=loadfile -s -v ./tests/
```

which tells pytest to:

- run as many test processes as they are CPU cores (which could be too many if you don't have a ton of RAM!)
- ensure that all tests from the same file will be run by the same test process
- do not capture output
- run in verbose mode



### Getting the list of all tests

All tests of the test suite:

```bash
pytest --collect-only -q
```

All tests of a given test file:

```bash
pytest tests/test_optimization.py --collect-only -q
```

### Run a specific test module

To run an individual test module:

```bash
pytest tests/utils/test_logging.py
```

### Run specific tests

Since unittest is used inside most of the tests, to run specific subtests you need to know the name of the unittest
class containing those tests. For example, it could be:

```bash
pytest tests/test_optimization.py::OptimizationTest::test_adam_w
```

Here:

- `tests/test_optimization.py` - the file with tests
- `OptimizationTest` - the name of the class
- `test_adam_w` - the name of the specific test function

If the file contains multiple classes, you can choose to run only tests of a given class. For example:

```bash
pytest tests/test_optimization.py::OptimizationTest
```

will run all the tests inside that class.

As mentioned earlier you can see what tests are contained inside the `OptimizationTest` class by running:

```bash
pytest tests/test_optimization.py::OptimizationTest --collect-only -q
```

You can run tests by keyword expressions.

To run only tests whose name contains `adam`:

```bash
pytest -k adam tests/test_optimization.py
```

Logical `and` and `or` can be used to indicate whether all keywords should match or either. `not` can be used to
negate.

To run all tests except those whose name contains `adam`:

```bash
pytest -k "not adam" tests/test_optimization.py
```

And you can combine the two patterns in one:

```bash
pytest -k "ada and not adam" tests/test_optimization.py
```

For example to run both `test_adafactor` and `test_adam_w` you can use:

```bash
pytest -k "test_adafactor or test_adam_w" tests/test_optimization.py
```

Note that we use `or` here, since we want either of the keywords to match to include both.

If you want to include only tests that include both patterns, `and` is to be used:

```bash
pytest -k "test and ada" tests/test_optimization.py
```

### Run `accelerate` tests

Sometimes you need to run `accelerate` tests on your models. For that you can just add `-m accelerate_tests` to your command, if let's say you want to run these tests on `OPT` run:

```bash
RUN_SLOW=1 pytest -m accelerate_tests tests/models/opt/test_modeling_opt.py
```


key: _node_content
string_value:
{"id_": "b25a3c49-43c8-4489-8f59-05275ebc3fff", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9daadbd-294f-4c73-bb06-bbfaf394994a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "48532a072b334e46759d814782e253d03273d5592441e8bbd0ffcd07b5453b79", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2d2640a-7706-4b30-af7d-97ac1306dd74", "node_type": "1", "metadata": {}, "hash": "f974b6090a54b5dbd7073ab95b8f0d8e220bfcb8ace9a39e6607283df322a180", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4736, "end_char_idx": 7874, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Run documentation tests

In order to test whether the documentation examples are correct, you should check that the `doctests` are passing.
As an example, let's use [`WhisperModel.forward`'s docstring](https://github.com/huggingface/transformers/blob/1124d95dbb1a3512d3e80791d73d0f541d1d7e9f/src/transformers/models/whisper/modeling_whisper.py#L1591-L1609)

```python
r"""
Returns:

Example:
    ```python
    >>> import torch
    >>> from transformers import WhisperModel, WhisperFeatureExtractor
    >>> from datasets import load_dataset

    >>> model = WhisperModel.from_pretrained("openai/whisper-base")
    >>> feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-base")
    >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
    >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt")
    >>> input_features = inputs.input_features
    >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id
    >>> last_hidden_state = model(input_features, decoder_input_ids=decoder_input_ids).last_hidden_state
    >>> list(last_hidden_state.shape)
    [1, 2, 512]
    ```"""

```

Just run the following line to automatically test every docstring example in the desired file:
```bash
pytest --doctest-modules <path_to_file_or_dir>
```
If the file has a markdown extention, you should add the `--doctest-glob="*.md"` argument.

### Run only modified tests

You can run the tests related to the unstaged files or the current branch (according to Git) by using [pytest-picked](https://github.com/anapaulagomes/pytest-picked). This is a great way of quickly testing your changes didn't break
anything, since it won't run the tests related to files you didn't touch.

```bash
pip install pytest-picked
```

```bash
pytest --picked
```

All tests will be run from files and folders which are modified, but not yet committed.

### Automatically rerun failed tests on source modification

[pytest-xdist](https://github.com/pytest-dev/pytest-xdist) provides a very useful feature of detecting all failed
tests, and then waiting for you to modify files and continuously re-rerun those failing tests until they pass while you
fix them. So that you don't need to re start pytest after you made the fix. This is repeated until all tests pass after
which again a full run is performed.

```bash
pip install pytest-xdist
```

To enter the mode: `pytest -f` or `pytest --looponfail`

File changes are detected by looking at `looponfailroots` root directories and all of their contents (recursively).
If the default for this value does not work for you, you can change it in your project by setting a configuration
option in `setup.cfg`:

```ini
[tool:pytest]
looponfailroots = transformers tests
```

or `pytest.ini`/``tox.ini`` files:

```ini
[pytest]
looponfailroots = transformers tests
```

This would lead to only looking for file changes in the respective directories, specified relatively to the ini-file’s
directory.

[pytest-watch](https://github.com/joeyespo/pytest-watch) is an alternative implementation of this functionality.


key: _node_content
string_value:
{"id_": "d2d2640a-7706-4b30-af7d-97ac1306dd74", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b25a3c49-43c8-4489-8f59-05275ebc3fff", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "1d716b2aaf47877b3d017f89e16002513dd5e05f95d3f9caae4a606bb0fda53b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a151706-38b0-439a-a113-88e92e1e8ef7", "node_type": "1", "metadata": {}, "hash": "917bb5093723d0630fe5e55dbd8341796a235f3296f05bbec98762e0850b3393", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7877, "end_char_idx": 9279, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Skip a test module

If you want to run all test modules, except a few you can exclude them by giving an explicit list of tests to run. For
example, to run all except `test_modeling_*.py` tests:

```bash
pytest *ls -1 tests/*py | grep -v test_modeling*
```

### Clearing state

CI builds and when isolation is important (against speed), cache should be cleared:

```bash
pytest --cache-clear tests
```

### Running tests in parallel

As mentioned earlier `make test` runs tests in parallel via `pytest-xdist` plugin (`-n X` argument, e.g. `-n 2`
to run 2 parallel jobs).

`pytest-xdist`'s `--dist=` option allows one to control how the tests are grouped. `--dist=loadfile` puts the
tests located in one file onto the same process.

Since the order of executed tests is different and unpredictable, if running the test suite with `pytest-xdist`
produces failures (meaning we have some undetected coupled tests), use [pytest-replay](https://github.com/ESSS/pytest-replay) to replay the tests in the same order, which should help with then somehow
reducing that failing sequence to a minimum.

### Test order and repetition

It's good to repeat the tests several times, in sequence, randomly, or in sets, to detect any potential
inter-dependency and state-related bugs (tear down). And the straightforward multiple repetition is just good to detect
some problems that get uncovered by randomness of DL.


key: _node_content
string_value:
{"id_": "2a151706-38b0-439a-a113-88e92e1e8ef7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2d2640a-7706-4b30-af7d-97ac1306dd74", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "15839530f15edfbee1c081d231be51ef00a8005c17d0cff25507a8a3f842badd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60cc3d58-1f86-4836-ba4a-2b2f26be26b6", "node_type": "1", "metadata": {}, "hash": "7a00815b5ed9a5e98bbbdd173f6c73689aad44eea906d406e7b59362c56f07de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7877, "end_char_idx": 11977, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Skip a test module

If you want to run all test modules, except a few you can exclude them by giving an explicit list of tests to run. For
example, to run all except `test_modeling_*.py` tests:

```bash
pytest *ls -1 tests/*py | grep -v test_modeling*
```

### Clearing state

CI builds and when isolation is important (against speed), cache should be cleared:

```bash
pytest --cache-clear tests
```

### Running tests in parallel

As mentioned earlier `make test` runs tests in parallel via `pytest-xdist` plugin (`-n X` argument, e.g. `-n 2`
to run 2 parallel jobs).

`pytest-xdist`'s `--dist=` option allows one to control how the tests are grouped. `--dist=loadfile` puts the
tests located in one file onto the same process.

Since the order of executed tests is different and unpredictable, if running the test suite with `pytest-xdist`
produces failures (meaning we have some undetected coupled tests), use [pytest-replay](https://github.com/ESSS/pytest-replay) to replay the tests in the same order, which should help with then somehow
reducing that failing sequence to a minimum.

### Test order and repetition

It's good to repeat the tests several times, in sequence, randomly, or in sets, to detect any potential
inter-dependency and state-related bugs (tear down). And the straightforward multiple repetition is just good to detect
some problems that get uncovered by randomness of DL.


#### Repeat tests

- [pytest-flakefinder](https://github.com/dropbox/pytest-flakefinder):

```bash
pip install pytest-flakefinder
```

And then run every test multiple times (50 by default):

```bash
pytest --flake-finder --flake-runs=5 tests/test_failing_test.py
```

<Tip>

This plugin doesn't work with `-n` flag from `pytest-xdist`.

</Tip>

<Tip>

There is another plugin `pytest-repeat`, but it doesn't work with `unittest`.

</Tip>

#### Run tests in a random order

```bash
pip install pytest-random-order
```

Important: the presence of `pytest-random-order` will automatically randomize tests, no configuration change or
command line options is required.

As explained earlier this allows detection of coupled tests - where one test's state affects the state of another. When
`pytest-random-order` is installed it will print the random seed it used for that session, e.g:

```bash
pytest tests
[...]
Using --random-order-bucket=module
Using --random-order-seed=573663
```

So that if the given particular sequence fails, you can reproduce it by adding that exact seed, e.g.:

```bash
pytest --random-order-seed=573663
[...]
Using --random-order-bucket=module
Using --random-order-seed=573663
```

It will only reproduce the exact order if you use the exact same list of tests (or no list at all). Once you start to
manually narrowing down the list you can no longer rely on the seed, but have to list them manually in the exact order
they failed and tell pytest to not randomize them instead using `--random-order-bucket=none`, e.g.:

```bash
pytest --random-order-bucket=none tests/test_a.py tests/test_c.py tests/test_b.py
```

To disable the shuffling for all tests:

```bash
pytest --random-order-bucket=none
```

By default `--random-order-bucket=module` is implied, which will shuffle the files on the module levels. It can also
shuffle on `class`, `package`, `global` and `none` levels. For the complete details please see its
[documentation](https://github.com/jbasko/pytest-random-order).

Another randomization alternative is: [`pytest-randomly`](https://github.com/pytest-dev/pytest-randomly). This
module has a very similar functionality/interface, but it doesn't have the bucket modes available in
`pytest-random-order`. It has the same problem of imposing itself once installed.

### Look and feel variations

#### pytest-sugar

[pytest-sugar](https://github.com/Frozenball/pytest-sugar) is a plugin that improves the look-n-feel, adds a
progressbar, and show tests that fail and the assert instantly. It gets activated automatically upon installation.

```bash
pip install pytest-sugar
```

To run tests without it, run:

```bash
pytest -p no:sugar
```

or uninstall it.


key: _node_content
string_value:
{"id_": "60cc3d58-1f86-4836-ba4a-2b2f26be26b6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a151706-38b0-439a-a113-88e92e1e8ef7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "b92131052334b4f4587b8fec26cdab30d2627fae78daa6d2aac44145d6ed5e46", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e7b44ac9-7e6a-49eb-ae8b-acfc1ae9ca71", "node_type": "1", "metadata": {}, "hash": "24f2914553df558d2923be8e790b04a4e9faf48c08ad450511382f210b06682f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11981, "end_char_idx": 14943, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Report each sub-test name and its progress

For a single or a group of tests via `pytest` (after `pip install pytest-pspec`):

```bash
pytest --pspec tests/test_optimization.py
```

#### Instantly shows failed tests

[pytest-instafail](https://github.com/pytest-dev/pytest-instafail) shows failures and errors instantly instead of
waiting until the end of test session.

```bash
pip install pytest-instafail
```

```bash
pytest --instafail
```

### To GPU or not to GPU

On a GPU-enabled setup, to test in CPU-only mode add `CUDA_VISIBLE_DEVICES=""`:

```bash
CUDA_VISIBLE_DEVICES="" pytest tests/utils/test_logging.py
```

or if you have multiple gpus, you can specify which one is to be used by `pytest`. For example, to use only the
second gpu if you have gpus `0` and `1`, you can run:

```bash
CUDA_VISIBLE_DEVICES="1" pytest tests/utils/test_logging.py
```

This is handy when you want to run different tasks on different GPUs.

Some tests must be run on CPU-only, others on either CPU or GPU or TPU, yet others on multiple-GPUs. The following skip
decorators are used to set the requirements of tests CPU/GPU/TPU-wise:

- `require_torch` - this test will run only under torch
- `require_torch_gpu` - as `require_torch` plus requires at least 1 GPU
- `require_torch_multi_gpu` - as `require_torch` plus requires at least 2 GPUs
- `require_torch_non_multi_gpu` - as `require_torch` plus requires 0 or 1 GPUs
- `require_torch_up_to_2_gpus` - as `require_torch` plus requires 0 or 1 or 2 GPUs
- `require_torch_xla` - as `require_torch` plus requires at least 1 TPU

Let's depict the GPU requirements in the following table:


| n gpus | decorator                      |
|--------|--------------------------------|
| `>= 0` | `@require_torch`               |
| `>= 1` | `@require_torch_gpu`           |
| `>= 2` | `@require_torch_multi_gpu`     |
| `< 2`  | `@require_torch_non_multi_gpu` |
| `< 3`  | `@require_torch_up_to_2_gpus`  |


For example, here is a test that must be run only when there are 2 or more GPUs available and pytorch is installed:

```python no-style
@require_torch_multi_gpu
def test_example_with_multi_gpu():
```

If a test requires `tensorflow` use the `require_tf` decorator. For example:

```python no-style
@require_tf
def test_tf_thing_with_tensorflow():
```

These decorators can be stacked. For example, if a test is slow and requires at least one GPU under pytorch, here is
how to set it up:

```python no-style
@require_torch_gpu
@slow
def test_example_slow_on_gpu():
```

Some decorators like `@parametrized` rewrite test names, therefore `@require_*` skip decorators have to be listed
last for them to work correctly. Here is an example of the correct usage:

```python no-style
@parameterized.expand(...)
@require_torch_multi_gpu
def test_integration_foo():
```

This order problem doesn't exist with `@pytest.mark.parametrize`, you can put it first or last and it will still
work. But it only works with non-unittests.


key: _node_content
string_value:
{"id_": "e7b44ac9-7e6a-49eb-ae8b-acfc1ae9ca71", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60cc3d58-1f86-4836-ba4a-2b2f26be26b6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "9630a06f8c6087651f172793f3bc8870a517a1a3d8cfdb661c18ea28c232585d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6578008d-e889-4a70-8c89-3c9dc9335cc8", "node_type": "1", "metadata": {}, "hash": "5c029f3134ee9441daac81eecd7c004fddadc9dff8b502bd6f64118c78f104e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13615, "end_char_idx": 16647, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
| n gpus | decorator                      |
|--------|--------------------------------|
| `>= 0` | `@require_torch`               |
| `>= 1` | `@require_torch_gpu`           |
| `>= 2` | `@require_torch_multi_gpu`     |
| `< 2`  | `@require_torch_non_multi_gpu` |
| `< 3`  | `@require_torch_up_to_2_gpus`  |


For example, here is a test that must be run only when there are 2 or more GPUs available and pytorch is installed:

```python no-style
@require_torch_multi_gpu
def test_example_with_multi_gpu():
```

If a test requires `tensorflow` use the `require_tf` decorator. For example:

```python no-style
@require_tf
def test_tf_thing_with_tensorflow():
```

These decorators can be stacked. For example, if a test is slow and requires at least one GPU under pytorch, here is
how to set it up:

```python no-style
@require_torch_gpu
@slow
def test_example_slow_on_gpu():
```

Some decorators like `@parametrized` rewrite test names, therefore `@require_*` skip decorators have to be listed
last for them to work correctly. Here is an example of the correct usage:

```python no-style
@parameterized.expand(...)
@require_torch_multi_gpu
def test_integration_foo():
```

This order problem doesn't exist with `@pytest.mark.parametrize`, you can put it first or last and it will still
work. But it only works with non-unittests.

Inside tests:

- How many GPUs are available:

```python
from transformers.testing_utils import get_gpu_count

n_gpu = get_gpu_count()  # works with torch and tf
```

### Testing with a specific PyTorch backend or device

To run the test suite on a specific torch device add `TRANSFORMERS_TEST_DEVICE="$device"` where `$device` is the target backend. For example, to test on CPU only:

```bash
TRANSFORMERS_TEST_DEVICE="cpu" pytest tests/utils/test_logging.py
```

This variable is useful for testing custom or less common PyTorch backends such as `mps`, `xpu` or `npu`. It can also be used to achieve the same effect as `CUDA_VISIBLE_DEVICES` by targeting specific GPUs or testing in CPU-only mode.

Certain devices will require an additional import after importing `torch` for the first time. This can be specified using the environment variable `TRANSFORMERS_TEST_BACKEND`:

```bash
TRANSFORMERS_TEST_BACKEND="torch_npu" pytest tests/utils/test_logging.py
```
Alternative backends may also require the replacement of device-specific functions. For example `torch.cuda.manual_seed` may need to be replaced with a device-specific seed setter like `torch.npu.manual_seed` or `torch.xpu.manual_seed` to correctly set a random seed on the device. To specify a new backend with backend-specific device functions when running the test suite, create a Python device specification file `spec.py` in the format:

```python
import torch
import torch_npu # for xpu, replace it with `import intel_extension_for_pytorch`
# !! Further additional imports can be added here !!

# Specify the device name (eg. 'cuda', 'cpu', 'npu', 'xpu', 'mps')
DEVICE_NAME = 'npu'

# Specify device-specific backends to dispatch to.


key: _node_content
string_value:
{"id_": "6578008d-e889-4a70-8c89-3c9dc9335cc8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e7b44ac9-7e6a-49eb-ae8b-acfc1ae9ca71", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "8c6e9061184fae0f0c1e70d4a1f433a49ba467517027c57abbc95a45f7a2a174", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d85d86a6-50df-4b25-abe8-18fdcf8009a4", "node_type": "1", "metadata": {}, "hash": "2a451f1fb0ffa1308309626b131e0faf8fed8ed244dea65a4d4670f8ca61da26", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14945, "end_char_idx": 18213, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Inside tests:

- How many GPUs are available:

```python
from transformers.testing_utils import get_gpu_count

n_gpu = get_gpu_count()  # works with torch and tf
```

### Testing with a specific PyTorch backend or device

To run the test suite on a specific torch device add `TRANSFORMERS_TEST_DEVICE="$device"` where `$device` is the target backend. For example, to test on CPU only:

```bash
TRANSFORMERS_TEST_DEVICE="cpu" pytest tests/utils/test_logging.py
```

This variable is useful for testing custom or less common PyTorch backends such as `mps`, `xpu` or `npu`. It can also be used to achieve the same effect as `CUDA_VISIBLE_DEVICES` by targeting specific GPUs or testing in CPU-only mode.

Certain devices will require an additional import after importing `torch` for the first time. This can be specified using the environment variable `TRANSFORMERS_TEST_BACKEND`:

```bash
TRANSFORMERS_TEST_BACKEND="torch_npu" pytest tests/utils/test_logging.py
```
Alternative backends may also require the replacement of device-specific functions. For example `torch.cuda.manual_seed` may need to be replaced with a device-specific seed setter like `torch.npu.manual_seed` or `torch.xpu.manual_seed` to correctly set a random seed on the device. To specify a new backend with backend-specific device functions when running the test suite, create a Python device specification file `spec.py` in the format:

```python
import torch
import torch_npu # for xpu, replace it with `import intel_extension_for_pytorch`
# !! Further additional imports can be added here !!

# Specify the device name (eg. 'cuda', 'cpu', 'npu', 'xpu', 'mps')
DEVICE_NAME = 'npu'

# Specify device-specific backends to dispatch to.
# If not specified, will fallback to 'default' in 'testing_utils.py`
MANUAL_SEED_FN = torch.npu.manual_seed
EMPTY_CACHE_FN = torch.npu.empty_cache
DEVICE_COUNT_FN = torch.npu.device_count
```
This format also allows for specification of any additional imports required. To use this file to replace equivalent methods in the test suite, set the environment variable `TRANSFORMERS_TEST_DEVICE_SPEC` to the path of the spec file, e.g. `TRANSFORMERS_TEST_DEVICE_SPEC=spec.py`.

Currently, only `MANUAL_SEED_FN`, `EMPTY_CACHE_FN` and `DEVICE_COUNT_FN` are supported for device-specific dispatch.

### Distributed training

`pytest` can't deal with distributed training directly. If this is attempted - the sub-processes don't do the right
thing and end up thinking they are `pytest` and start running the test suite in loops. It works, however, if one
spawns a normal process that then spawns off multiple workers and manages the IO pipes.

Here are some tests that use it:

- [test_trainer_distributed.py](https://github.com/huggingface/transformers/tree/main/tests/trainer/test_trainer_distributed.py)
- [test_deepspeed.py](https://github.com/huggingface/transformers/tree/main/tests/deepspeed/test_deepspeed.py)

To jump right into the execution point, search for the `execute_subprocess_async` call in those tests.

You will need at least 2 GPUs to see these tests in action:

```bash
CUDA_VISIBLE_DEVICES=0,1 RUN_SLOW=1 pytest -sv tests/test_trainer_distributed.py
```

### Output capture

During test execution any output sent to `stdout` and `stderr` is captured.


key: _node_content
string_value:
{"id_": "d85d86a6-50df-4b25-abe8-18fdcf8009a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6578008d-e889-4a70-8c89-3c9dc9335cc8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "eb97c9f76f69b1f6e7a04fedae66804bf477ffb1cd1f7e1e586dca50518ad21a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c3284774-3bd7-4b83-b5bf-29a2595fa715", "node_type": "1", "metadata": {}, "hash": "0af4d5c556c3125733e710f7efee3f543a79d1783a9ee4190ae684f181d7d7bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16648, "end_char_idx": 19928, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# If not specified, will fallback to 'default' in 'testing_utils.py`
MANUAL_SEED_FN = torch.npu.manual_seed
EMPTY_CACHE_FN = torch.npu.empty_cache
DEVICE_COUNT_FN = torch.npu.device_count
```
This format also allows for specification of any additional imports required. To use this file to replace equivalent methods in the test suite, set the environment variable `TRANSFORMERS_TEST_DEVICE_SPEC` to the path of the spec file, e.g. `TRANSFORMERS_TEST_DEVICE_SPEC=spec.py`.

Currently, only `MANUAL_SEED_FN`, `EMPTY_CACHE_FN` and `DEVICE_COUNT_FN` are supported for device-specific dispatch.

### Distributed training

`pytest` can't deal with distributed training directly. If this is attempted - the sub-processes don't do the right
thing and end up thinking they are `pytest` and start running the test suite in loops. It works, however, if one
spawns a normal process that then spawns off multiple workers and manages the IO pipes.

Here are some tests that use it:

- [test_trainer_distributed.py](https://github.com/huggingface/transformers/tree/main/tests/trainer/test_trainer_distributed.py)
- [test_deepspeed.py](https://github.com/huggingface/transformers/tree/main/tests/deepspeed/test_deepspeed.py)

To jump right into the execution point, search for the `execute_subprocess_async` call in those tests.

You will need at least 2 GPUs to see these tests in action:

```bash
CUDA_VISIBLE_DEVICES=0,1 RUN_SLOW=1 pytest -sv tests/test_trainer_distributed.py
```

### Output capture

During test execution any output sent to `stdout` and `stderr` is captured. If a test or a setup method fails, its
according captured output will usually be shown along with the failure traceback.

To disable output capturing and to get the `stdout` and `stderr` normally, use `-s` or `--capture=no`:

```bash
pytest -s tests/utils/test_logging.py
```

To send test results to JUnit format output:

```bash
pytest tests --junitxml=result.xml
```

### Color control

To have no color (e.g., yellow on white background is not readable):

```bash
pytest --color=no tests/utils/test_logging.py
```

### Sending test report to online pastebin service

Creating a URL for each test failure:

```bash
pytest --pastebin=failed tests/utils/test_logging.py
```

This will submit test run information to a remote Paste service and provide a URL for each failure. You may select
tests as usual or add for example -x if you only want to send one particular failure.

Creating a URL for a whole test session log:

```bash
pytest --pastebin=all tests/utils/test_logging.py
```

## Writing tests

🤗 transformers tests are based on `unittest`, but run by `pytest`, so most of the time features from both systems
can be used.

You can read [here](https://docs.pytest.org/en/stable/unittest.html) which features are supported, but the important
thing to remember is that most `pytest` fixtures don't work. Neither parametrization, but we use the module
`parameterized` that works in a similar way.


### Parametrization

Often, there is a need to run the same test multiple times, but with different arguments. It could be done from within
the test, but then there is no way of running that test for just one set of arguments.

```python
# test_this1.py
import unittest
from parameterized import parameterized


key: _node_content
string_value:
{"id_": "c3284774-3bd7-4b83-b5bf-29a2595fa715", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d85d86a6-50df-4b25-abe8-18fdcf8009a4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "fe85ddc805e58b8f83103e2b2b9b9bccb4a6517d2b5b20ef6d510811022a80d4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f3965dc6-d4a7-4504-8563-6486620f5ee7", "node_type": "1", "metadata": {}, "hash": "d9b43fd7e002a4234d0ca023a2dbd2e534dfbe13b6e5b3af3fee55d81ab82fd6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18214, "end_char_idx": 21695, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If a test or a setup method fails, its
according captured output will usually be shown along with the failure traceback.

To disable output capturing and to get the `stdout` and `stderr` normally, use `-s` or `--capture=no`:

```bash
pytest -s tests/utils/test_logging.py
```

To send test results to JUnit format output:

```bash
pytest tests --junitxml=result.xml
```

### Color control

To have no color (e.g., yellow on white background is not readable):

```bash
pytest --color=no tests/utils/test_logging.py
```

### Sending test report to online pastebin service

Creating a URL for each test failure:

```bash
pytest --pastebin=failed tests/utils/test_logging.py
```

This will submit test run information to a remote Paste service and provide a URL for each failure. You may select
tests as usual or add for example -x if you only want to send one particular failure.

Creating a URL for a whole test session log:

```bash
pytest --pastebin=all tests/utils/test_logging.py
```

## Writing tests

🤗 transformers tests are based on `unittest`, but run by `pytest`, so most of the time features from both systems
can be used.

You can read [here](https://docs.pytest.org/en/stable/unittest.html) which features are supported, but the important
thing to remember is that most `pytest` fixtures don't work. Neither parametrization, but we use the module
`parameterized` that works in a similar way.


### Parametrization

Often, there is a need to run the same test multiple times, but with different arguments. It could be done from within
the test, but then there is no way of running that test for just one set of arguments.

```python
# test_this1.py
import unittest
from parameterized import parameterized


class TestMathUnitTest(unittest.TestCase):
    @parameterized.expand(
        [
            ("negative", -1.5, -2.0),
            ("integer", 1, 1.0),
            ("large fraction", 1.6, 1),
        ]
    )
    def test_floor(self, name, input, expected):
        assert_equal(math.floor(input), expected)
```

Now, by default this test will be run 3 times, each time with the last 3 arguments of `test_floor` being assigned the
corresponding arguments in the parameter list.

and you could run just the `negative` and `integer` sets of params with:

```bash
pytest -k "negative and integer" tests/test_mytest.py
```

or all but `negative` sub-tests, with:

```bash
pytest -k "not negative" tests/test_mytest.py
```

Besides using the `-k` filter that was just mentioned, you can find out the exact name of each sub-test and run any
or all of them using their exact names.

```bash
pytest test_this1.py --collect-only -q
```

and it will list:

```bash
test_this1.py::TestMathUnitTest::test_floor_0_negative
test_this1.py::TestMathUnitTest::test_floor_1_integer
test_this1.py::TestMathUnitTest::test_floor_2_large_fraction
```

So now you can run just 2 specific sub-tests:

```bash
pytest test_this1.py::TestMathUnitTest::test_floor_0_negative  test_this1.py::TestMathUnitTest::test_floor_1_integer
```

The module [parameterized](https://pypi.org/project/parameterized/) which is already in the developer dependencies
of `transformers` works for both: `unittests` and `pytest` tests.

If, however, the test is not a `unittest`, you may use `pytest.mark.parametrize` (or you may see it being used in
some existing tests, mostly under `examples`).

Here is the same example, this time using `pytest`'s `parametrize` marker:

```python
# test_this2.py
import pytest


key: _node_content
string_value:
{"id_": "f3965dc6-d4a7-4504-8563-6486620f5ee7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c3284774-3bd7-4b83-b5bf-29a2595fa715", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "56dae991ce399fa3e168a73a26cf6a6615a455976b63dfe33015915b850a2dcb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "00ad1d1a-2fc9-4f2d-ae8a-a3ac6289de8f", "node_type": "1", "metadata": {}, "hash": "e1a9f85e7d761ccd28d2f659c8ec5ad1997f84394b3c97b38fde0f1b736b7a44", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21698, "end_char_idx": 24359, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
@pytest.mark.parametrize(
    "name, input, expected",
    [
        ("negative", -1.5, -2.0),
        ("integer", 1, 1.0),
        ("large fraction", 1.6, 1),
    ],
)
def test_floor(name, input, expected):
    assert_equal(math.floor(input), expected)
```

Same as with `parameterized`, with `pytest.mark.parametrize` you can have a fine control over which sub-tests are
run, if the `-k` filter doesn't do the job. Except, this parametrization function creates a slightly different set of
names for the sub-tests. Here is what they look like:

```bash
pytest test_this2.py --collect-only -q
```

and it will list:

```bash
test_this2.py::test_floor[integer-1-1.0]
test_this2.py::test_floor[negative--1.5--2.0]
test_this2.py::test_floor[large fraction-1.6-1]
```

So now you can run just the specific test:

```bash
pytest test_this2.py::test_floor[negative--1.5--2.0] test_this2.py::test_floor[integer-1-1.0]
```

as in the previous example.



### Files and directories

In tests often we need to know where things are relative to the current test file, and it's not trivial since the test
could be invoked from more than one directory or could reside in sub-directories with different depths. A helper class
`transformers.test_utils.TestCasePlus` solves this problem by sorting out all the basic paths and provides easy
accessors to them:

- `pathlib` objects (all fully resolved):

  - `test_file_path` - the current test file path, i.e. `__file__`
  - `test_file_dir` - the directory containing the current test file
  - `tests_dir` - the directory of the `tests` test suite
  - `examples_dir` - the directory of the `examples` test suite
  - `repo_root_dir` - the directory of the repository
  - `src_dir` - the directory of `src` (i.e. where the `transformers` sub-dir resides)

- stringified paths---same as above but these return paths as strings, rather than `pathlib` objects:

  - `test_file_path_str`
  - `test_file_dir_str`
  - `tests_dir_str`
  - `examples_dir_str`
  - `repo_root_dir_str`
  - `src_dir_str`

To start using those all you need is to make sure that the test resides in a subclass of
`transformers.test_utils.TestCasePlus`. For example:

```python
from transformers.testing_utils import TestCasePlus


class PathExampleTest(TestCasePlus):
    def test_something_involving_local_locations(self):
        data_dir = self.tests_dir / "fixtures/tests_samples/wmt_en_ro"
```

If you don't need to manipulate paths via `pathlib` or you just need a path as a string, you can always invoked
`str()` on the `pathlib` object or use the accessors ending with `_str`. For example:

```python
from transformers.testing_utils import TestCasePlus


key: _node_content
string_value:
{"id_": "00ad1d1a-2fc9-4f2d-ae8a-a3ac6289de8f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f3965dc6-d4a7-4504-8563-6486620f5ee7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "f1dd6cdbb2a278da0975452a71e754466ce7f0749afa6f59d320418cc5fbab81", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed1c35c3-02c6-4eaa-8e17-817e5e7167d9", "node_type": "1", "metadata": {}, "hash": "2fb931b43c9ad8db93408479f62947c456c4199844cd902abdc43be80ef1ecba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23930, "end_char_idx": 27256, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class PathExampleTest(TestCasePlus):
    def test_something_involving_local_locations(self):
        data_dir = self.tests_dir / "fixtures/tests_samples/wmt_en_ro"
```

If you don't need to manipulate paths via `pathlib` or you just need a path as a string, you can always invoked
`str()` on the `pathlib` object or use the accessors ending with `_str`. For example:

```python
from transformers.testing_utils import TestCasePlus


class PathExampleTest(TestCasePlus):
    def test_something_involving_stringified_locations(self):
        examples_dir = self.examples_dir_str
```

### Temporary files and directories

Using unique temporary files and directories are essential for parallel test running, so that the tests won't overwrite
each other's data. Also we want to get the temporary files and directories removed at the end of each test that created
them. Therefore, using packages like `tempfile`, which address these needs is essential.

However, when debugging tests, you need to be able to see what goes into the temporary file or directory and you want
to know it's exact path and not having it randomized on every test re-run.

A helper class `transformers.test_utils.TestCasePlus` is best used for such purposes. It's a sub-class of
`unittest.TestCase`, so we can easily inherit from it in the test modules.

Here is an example of its usage:

```python
from transformers.testing_utils import TestCasePlus


class ExamplesTests(TestCasePlus):
    def test_whatever(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
```

This code creates a unique temporary directory, and sets `tmp_dir` to its location.

- Create a unique temporary dir:

```python
def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir()
```

`tmp_dir` will contain the path to the created temporary dir. It will be automatically removed at the end of the
test.

- Create a temporary dir of my choice, ensure it's empty before the test starts and don't empty it after the test.

```python
def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir("./xxx")
```

This is useful for debug when you want to monitor a specific directory and want to make sure the previous tests didn't
leave any data in there.

- You can override the default behavior by directly overriding the `before` and `after` args, leading to one of the
  following behaviors:

  - `before=True`: the temporary dir will always be cleared at the beginning of the test.
  - `before=False`: if the temporary dir already existed, any existing files will remain there.
  - `after=True`: the temporary dir will always be deleted at the end of the test.
  - `after=False`: the temporary dir will always be left intact at the end of the test.

<Tip>

In order to run the equivalent of `rm -r` safely, only subdirs of the project repository checkout are allowed if
an explicit `tmp_dir` is used, so that by mistake no `/tmp` or similar important part of the filesystem will
get nuked. i.e. please always pass paths that start with `./`.

</Tip>

<Tip>

Each test can register multiple temporary directories and they all will get auto-removed, unless requested
otherwise.

</Tip>

### Temporary sys.path override

If you need to temporary override `sys.path` to import from another test for example, you can use the
`ExtendSysPath` context manager. Example:


key: _node_content
string_value:
{"id_": "ed1c35c3-02c6-4eaa-8e17-817e5e7167d9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "00ad1d1a-2fc9-4f2d-ae8a-a3ac6289de8f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "785e470932aa0e0ddd1126804a378e1b28852fb62818ee7e3118bd13a9e56ec1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "61406dd6-ed3c-4205-a007-f91b675d5e37", "node_type": "1", "metadata": {}, "hash": "74d4c3e2ea345f79bf0dbdf2c36584b8b9798f3ded4c59ca4c7ecd01471c3a58", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27259, "end_char_idx": 30289, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import os
from transformers.testing_utils import ExtendSysPath

bindir = os.path.abspath(os.path.dirname(__file__))
with ExtendSysPath(f"{bindir}/.."):
    from test_trainer import TrainerIntegrationCommon  # noqa
```

### Skipping tests

This is useful when a bug is found and a new test is written, yet the bug is not fixed yet. In order to be able to
commit it to the main repository we need make sure it's skipped during `make test`.

Methods:

-  A **skip** means that you expect your test to pass only if some conditions are met, otherwise pytest should skip
  running the test altogether. Common examples are skipping windows-only tests on non-windows platforms, or skipping
  tests that depend on an external resource which is not available at the moment (for example a database).

-  A **xfail** means that you expect a test to fail for some reason. A common example is a test for a feature not yet
  implemented, or a bug not yet fixed. When a test passes despite being expected to fail (marked with
  pytest.mark.xfail), it’s an xpass and will be reported in the test summary.

One of the important differences between the two is that `skip` doesn't run the test, and `xfail` does. So if the
code that's buggy causes some bad state that will affect other tests, do not use `xfail`.

#### Implementation

- Here is how to skip whole test unconditionally:

```python no-style
@unittest.skip(reason="this bug needs to be fixed")
def test_feature_x():
```

or via pytest:

```python no-style
@pytest.mark.skip(reason="this bug needs to be fixed")
```

or the `xfail` way:

```python no-style
@pytest.mark.xfail
def test_feature_x():
```


Here's how to skip a test based on internal checks within the test:

```python
def test_feature_x():
    if not has_something():
        pytest.skip("unsupported configuration")
```

or the whole module:

```python
import pytest

if not pytest.config.getoption("--custom-flag"):
    pytest.skip("--custom-flag is missing, skipping tests", allow_module_level=True)
```

or the `xfail` way:

```python
def test_feature_x():
    pytest.xfail("expected to fail until bug XYZ is fixed")
```

- Here is how to skip all tests in a module if some import is missing:

```python
docutils = pytest.importorskip("docutils", minversion="0.3")
```

-  Skip a test based on a condition:

```python no-style
@pytest.mark.skipif(sys.version_info < (3,6), reason="requires python3.6 or higher")
def test_feature_x():
```

or:

```python no-style
@unittest.skipIf(torch_device == "cpu", "Can't do half precision")
def test_feature_x():
```

or skip the whole module:

```python no-style
@pytest.mark.skipif(sys.platform == 'win32', reason="does not run on windows")
class TestClass():
    def test_feature_x(self):
```

More details, example and ways are [here](https://docs.pytest.org/en/latest/skipping.html).

### Slow tests

The library of tests is ever-growing, and some of the tests take minutes to run, therefore we can't afford waiting for
an hour for the test suite to complete on CI.


key: _node_content
string_value:
{"id_": "61406dd6-ed3c-4205-a007-f91b675d5e37", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ed1c35c3-02c6-4eaa-8e17-817e5e7167d9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "350099125ec5f4ee16eb20324a9559d53b3bb6704d7ab457d9595af2f90c2fe0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a9ec9e1-fb06-41db-9d98-d44426164fa0", "node_type": "1", "metadata": {}, "hash": "68842981d9d4c92acafa2028312c1a402e6a9f2898f5b65b69f4b1caaa5936d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28915, "end_char_idx": 32167, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here's how to skip a test based on internal checks within the test:

```python
def test_feature_x():
    if not has_something():
        pytest.skip("unsupported configuration")
```

or the whole module:

```python
import pytest

if not pytest.config.getoption("--custom-flag"):
    pytest.skip("--custom-flag is missing, skipping tests", allow_module_level=True)
```

or the `xfail` way:

```python
def test_feature_x():
    pytest.xfail("expected to fail until bug XYZ is fixed")
```

- Here is how to skip all tests in a module if some import is missing:

```python
docutils = pytest.importorskip("docutils", minversion="0.3")
```

-  Skip a test based on a condition:

```python no-style
@pytest.mark.skipif(sys.version_info < (3,6), reason="requires python3.6 or higher")
def test_feature_x():
```

or:

```python no-style
@unittest.skipIf(torch_device == "cpu", "Can't do half precision")
def test_feature_x():
```

or skip the whole module:

```python no-style
@pytest.mark.skipif(sys.platform == 'win32', reason="does not run on windows")
class TestClass():
    def test_feature_x(self):
```

More details, example and ways are [here](https://docs.pytest.org/en/latest/skipping.html).

### Slow tests

The library of tests is ever-growing, and some of the tests take minutes to run, therefore we can't afford waiting for
an hour for the test suite to complete on CI. Therefore, with some exceptions for essential tests, slow tests should be
marked as in the example below:

```python no-style
from transformers.testing_utils import slow
@slow
def test_integration_foo():
```

Once a test is marked as `@slow`, to run such tests set `RUN_SLOW=1` env var, e.g.:

```bash
RUN_SLOW=1 pytest tests
```

Some decorators like `@parameterized` rewrite test names, therefore `@slow` and the rest of the skip decorators
`@require_*` have to be listed last for them to work correctly. Here is an example of the correct usage:

```python no-style
@parameterized.expand(...)
@slow
def test_integration_foo():
```

As explained at the beginning of this document, slow tests get to run on a scheduled basis, rather than in PRs CI
checks. So it's possible that some problems will be missed during a PR submission and get merged. Such problems will
get caught during the next scheduled CI job. But it also means that it's important to run the slow tests on your
machine before submitting the PR.

Here is a rough decision making mechanism for choosing which tests should be marked as slow:

If the test is focused on one of the library's internal components (e.g., modeling files, tokenization files,
pipelines), then we should run that test in the non-slow test suite. If it's focused on an other aspect of the library,
such as the documentation or the examples, then we should run these tests in the slow test suite. And then, to refine
this approach we should have exceptions:

- All tests that need to download a heavy set of weights or a dataset that is larger than ~50MB (e.g., model or
  tokenizer integration tests, pipeline integration tests) should be set to slow. If you're adding a new model, you
  should create and upload to the hub a tiny version of it (with random weights) for integration tests. This is
  discussed in the following paragraphs.


key: _node_content
string_value:
{"id_": "0a9ec9e1-fb06-41db-9d98-d44426164fa0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "61406dd6-ed3c-4205-a007-f91b675d5e37", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "dd0b9532c6290bec38ddff434e71509980b1ed00054371652439ceb2f1199161", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a84292c4-c89a-4190-bd9a-83478579bedc", "node_type": "1", "metadata": {}, "hash": "0007f7e2b27124ebd7f88da8ae267c5696e33e0ac98446b4dfa78b0544a036e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30581, "end_char_idx": 33852, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
:

```bash
RUN_SLOW=1 pytest tests
```

Some decorators like `@parameterized` rewrite test names, therefore `@slow` and the rest of the skip decorators
`@require_*` have to be listed last for them to work correctly. Here is an example of the correct usage:

```python no-style
@parameterized.expand(...)
@slow
def test_integration_foo():
```

As explained at the beginning of this document, slow tests get to run on a scheduled basis, rather than in PRs CI
checks. So it's possible that some problems will be missed during a PR submission and get merged. Such problems will
get caught during the next scheduled CI job. But it also means that it's important to run the slow tests on your
machine before submitting the PR.

Here is a rough decision making mechanism for choosing which tests should be marked as slow:

If the test is focused on one of the library's internal components (e.g., modeling files, tokenization files,
pipelines), then we should run that test in the non-slow test suite. If it's focused on an other aspect of the library,
such as the documentation or the examples, then we should run these tests in the slow test suite. And then, to refine
this approach we should have exceptions:

- All tests that need to download a heavy set of weights or a dataset that is larger than ~50MB (e.g., model or
  tokenizer integration tests, pipeline integration tests) should be set to slow. If you're adding a new model, you
  should create and upload to the hub a tiny version of it (with random weights) for integration tests. This is
  discussed in the following paragraphs.
- All tests that need to do a training not specifically optimized to be fast should be set to slow.
- We can introduce exceptions if some of these should-be-non-slow tests are excruciatingly slow, and set them to
  `@slow`. Auto-modeling tests, which save and load large files to disk, are a good example of tests that are marked
  as `@slow`.
- If a test completes under 1 second on CI (including downloads if any) then it should be a normal test regardless.

Collectively, all the non-slow tests need to cover entirely the different internals, while remaining fast. For example,
a significant coverage can be achieved by testing with specially created tiny models with random weights. Such models
have the very minimal number of layers (e.g., 2), vocab size (e.g., 1000), etc. Then the `@slow` tests can use large
slow models to do qualitative testing. To see the use of these simply look for *tiny* models with:

```bash
grep tiny tests examples
```

Here is an example of a [script](https://github.com/huggingface/transformers/tree/main/scripts/fsmt/fsmt-make-tiny-model.py) that created the tiny model
[stas/tiny-wmt19-en-de](https://huggingface.co/stas/tiny-wmt19-en-de). You can easily adjust it to your specific
model's architecture.

It's easy to measure the run-time incorrectly if for example there is an overheard of downloading a huge model, but if
you test it locally the downloaded files would be cached and thus the download time not measured. Hence check the
execution speed report in CI logs instead (the output of `pytest --durations=0 tests`).

That report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast.


key: _node_content
string_value:
{"id_": "a84292c4-c89a-4190-bd9a-83478579bedc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a9ec9e1-fb06-41db-9d98-d44426164fa0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "a507259697627049e8ed01ab7da2dbeb57905c35522d354599cc216a1df85c92", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "945ec437-8e66-4012-b5d1-ec9ac2e4e30b", "node_type": "1", "metadata": {}, "hash": "bb8fd2419b90aa9446a4b7ceda7da957800c8e340f9d0aa618d7dc27f5e51d88", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32268, "end_char_idx": 35312, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- We can introduce exceptions if some of these should-be-non-slow tests are excruciatingly slow, and set them to
  `@slow`. Auto-modeling tests, which save and load large files to disk, are a good example of tests that are marked
  as `@slow`.
- If a test completes under 1 second on CI (including downloads if any) then it should be a normal test regardless.

Collectively, all the non-slow tests need to cover entirely the different internals, while remaining fast. For example,
a significant coverage can be achieved by testing with specially created tiny models with random weights. Such models
have the very minimal number of layers (e.g., 2), vocab size (e.g., 1000), etc. Then the `@slow` tests can use large
slow models to do qualitative testing. To see the use of these simply look for *tiny* models with:

```bash
grep tiny tests examples
```

Here is an example of a [script](https://github.com/huggingface/transformers/tree/main/scripts/fsmt/fsmt-make-tiny-model.py) that created the tiny model
[stas/tiny-wmt19-en-de](https://huggingface.co/stas/tiny-wmt19-en-de). You can easily adjust it to your specific
model's architecture.

It's easy to measure the run-time incorrectly if for example there is an overheard of downloading a huge model, but if
you test it locally the downloaded files would be cached and thus the download time not measured. Hence check the
execution speed report in CI logs instead (the output of `pytest --durations=0 tests`).

That report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast.
If you notice that the test suite starts getting slow on CI, the top listing of this report will show the slowest
tests.


### Testing the stdout/stderr output

In order to test functions that write to `stdout` and/or `stderr`, the test can access those streams using the
`pytest`'s [capsys system](https://docs.pytest.org/en/latest/capture.html). Here is how this is accomplished:

```python
import sys


def print_to_stdout(s):
    print(s)


def print_to_stderr(s):
    sys.stderr.write(s)


def test_result_and_stdout(capsys):
    msg = "Hello"
    print_to_stdout(msg)
    print_to_stderr(msg)
    out, err = capsys.readouterr()  # consume the captured output streams
    # optional: if you want to replay the consumed streams:
    sys.stdout.write(out)
    sys.stderr.write(err)
    # test:
    assert msg in out
    assert msg in err
```

And, of course, most of the time, `stderr` will come as a part of an exception, so try/except has to be used in such
a case:

```python
def raise_exception(msg):
    raise ValueError(msg)


def test_something_exception():
    msg = "Not a good value"
    error = ""
    try:
        raise_exception(msg)
    except Exception as e:
        error = str(e)
        assert msg in error, f"{msg} is in the exception:\n{error}"
```

Another approach to capturing stdout is via `contextlib.redirect_stdout`:

```python
from io import StringIO
from contextlib import redirect_stdout


def print_to_stdout(s):
    print(s)


key: _node_content
string_value:
{"id_": "945ec437-8e66-4012-b5d1-ec9ac2e4e30b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a84292c4-c89a-4190-bd9a-83478579bedc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "455c5379d683369fb5a5318cf1e2330610b2a2424a2752b07cbc45d715223089", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b3f9c183-8d8e-462c-b095-b3d650d926d5", "node_type": "1", "metadata": {}, "hash": "2edc364b2b73fe64690d6423c8613550bd19d2662997787614084d0db44a5db2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33733, "end_char_idx": 37206, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
That report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast.
If you notice that the test suite starts getting slow on CI, the top listing of this report will show the slowest
tests.


### Testing the stdout/stderr output

In order to test functions that write to `stdout` and/or `stderr`, the test can access those streams using the
`pytest`'s [capsys system](https://docs.pytest.org/en/latest/capture.html). Here is how this is accomplished:

```python
import sys


def print_to_stdout(s):
    print(s)


def print_to_stderr(s):
    sys.stderr.write(s)


def test_result_and_stdout(capsys):
    msg = "Hello"
    print_to_stdout(msg)
    print_to_stderr(msg)
    out, err = capsys.readouterr()  # consume the captured output streams
    # optional: if you want to replay the consumed streams:
    sys.stdout.write(out)
    sys.stderr.write(err)
    # test:
    assert msg in out
    assert msg in err
```

And, of course, most of the time, `stderr` will come as a part of an exception, so try/except has to be used in such
a case:

```python
def raise_exception(msg):
    raise ValueError(msg)


def test_something_exception():
    msg = "Not a good value"
    error = ""
    try:
        raise_exception(msg)
    except Exception as e:
        error = str(e)
        assert msg in error, f"{msg} is in the exception:\n{error}"
```

Another approach to capturing stdout is via `contextlib.redirect_stdout`:

```python
from io import StringIO
from contextlib import redirect_stdout


def print_to_stdout(s):
    print(s)


def test_result_and_stdout():
    msg = "Hello"
    buffer = StringIO()
    with redirect_stdout(buffer):
        print_to_stdout(msg)
    out = buffer.getvalue()
    # optional: if you want to replay the consumed streams:
    sys.stdout.write(out)
    # test:
    assert msg in out
```

An important potential issue with capturing stdout is that it may contain `\r` characters that in normal `print`
reset everything that has been printed so far. There is no problem with `pytest`, but with `pytest -s` these
characters get included in the buffer, so to be able to have the test run with and without `-s`, you have to make an
extra cleanup to the captured output, using `re.sub(r'~.*\r', '', buf, 0, re.M)`.

But, then we have a helper context manager wrapper to automatically take care of it all, regardless of whether it has
some `\r`'s in it or not, so it's a simple:

```python
from transformers.testing_utils import CaptureStdout

with CaptureStdout() as cs:
    function_that_writes_to_stdout()
print(cs.out)
```

Here is a full test example:

```python
from transformers.testing_utils import CaptureStdout

msg = "Secret message\r"
final = "Hello World"
with CaptureStdout() as cs:
    print(msg + final)
assert cs.out == final + "\n", f"captured: {cs.out}, expecting {final}"
```

If you'd like to capture `stderr` use the `CaptureStderr` class instead:

```python
from transformers.testing_utils import CaptureStderr

with CaptureStderr() as cs:
    function_that_writes_to_stderr()
print(cs.err)
```

If you need to capture both streams at once, use the parent `CaptureStd` class:

```python
from transformers.testing_utils import CaptureStd

with CaptureStd() as cs:
    function_that_writes_to_stdout_and_stderr()
print(cs.err, cs.out)
```

Also, to aid debugging test issues, by default these context managers automatically replay the captured streams on exit
from the context.


key: _node_content
string_value:
{"id_": "b3f9c183-8d8e-462c-b095-b3d650d926d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "945ec437-8e66-4012-b5d1-ec9ac2e4e30b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "66c89b62c0728a74f94b513649d80397d0104a66df675d0f6be30811f8000335", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3efa7ead-a0f0-4f45-92d2-a95a49948ed4", "node_type": "1", "metadata": {}, "hash": "f45734368c77a8ef8f506540e9f9f591146d3397d76095912bb02705661f4375", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37209, "end_char_idx": 40340, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Capturing logger stream

If you need to validate the output of a logger, you can use `CaptureLogger`:

```python
from transformers import logging
from transformers.testing_utils import CaptureLogger

msg = "Testing 1, 2, 3"
logging.set_verbosity_info()
logger = logging.get_logger("transformers.models.bart.tokenization_bart")
with CaptureLogger(logger) as cl:
    logger.info(msg)
assert cl.out, msg + "\n"
```

### Testing with environment variables

If you want to test the impact of environment variables for a specific test you can use a helper decorator
`transformers.testing_utils.mockenv`

```python
from transformers.testing_utils import mockenv


class HfArgumentParserTest(unittest.TestCase):
    @mockenv(TRANSFORMERS_VERBOSITY="error")
    def test_env_override(self):
        env_level_str = os.getenv("TRANSFORMERS_VERBOSITY", None)
```

At times an external program needs to be called, which requires setting `PYTHONPATH` in `os.environ` to include
multiple local paths. A helper class `transformers.test_utils.TestCasePlus` comes to help:

```python
from transformers.testing_utils import TestCasePlus


class EnvExampleTest(TestCasePlus):
    def test_external_prog(self):
        env = self.get_env()
        # now call the external program, passing `env` to it
```

Depending on whether the test file was under the `tests` test suite or `examples` it'll correctly set up
`env[PYTHONPATH]` to include one of these two directories, and also the `src` directory to ensure the testing is
done against the current repo, and finally with whatever `env[PYTHONPATH]` was already set to before the test was
called if anything.

This helper method creates a copy of the `os.environ` object, so the original remains intact.


### Getting reproducible results

In some situations you may want to remove randomness for your tests. To get identical reproducible results set, you
will need to fix the seed:

```python
seed = 42

# python RNG
import random

random.seed(seed)

# pytorch RNGs
import torch

torch.manual_seed(seed)
torch.backends.cudnn.deterministic = True
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)

# numpy RNG
import numpy as np

np.random.seed(seed)

# tf RNG
tf.random.set_seed(seed)
```

### Debugging tests

To start a debugger at the point of the warning, do this:

```bash
pytest tests/utils/test_logging.py -W error::UserWarning --pdb
```

## Working with github actions workflows

To trigger a self-push workflow CI job, you must:

1. Create a new branch on `transformers` origin (not a fork!).
2. The branch name has to start with either `ci_` or `ci-` (`main` triggers it too, but we can't do PRs on
   `main`). It also gets triggered only for specific paths - you can find the up-to-date definition in case it
   changed since this document has been written [here](https://github.com/huggingface/transformers/blob/main/.github/workflows/self-push.yml) under *push:*
3. Create a PR from this branch.
4. Then you can see the job appear [here](https://github.com/huggingface/transformers/actions/workflows/self-push.yml). It may not run right away if there
   is a backlog.


key: _node_content
string_value:
{"id_": "3efa7ead-a0f0-4f45-92d2-a95a49948ed4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eb25e5f-8b07-473e-b5cc-2fc2660a2a8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "c1be3b5104cc4db1b5837cada4665bbcc1ee39853840405c67ae4e6bd0e0a8d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b3f9c183-8d8e-462c-b095-b3d650d926d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing", "title": "Testing", "tokens": 10586, "retrieve_doc": false, "source": "transformers"}, "hash": "81f33db00fb6765d78972019429bb2887923db4b9c9c493fb06d5fc550e45803", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38948, "end_char_idx": 43536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Getting reproducible results

In some situations you may want to remove randomness for your tests. To get identical reproducible results set, you
will need to fix the seed:

```python
seed = 42

# python RNG
import random

random.seed(seed)

# pytorch RNGs
import torch

torch.manual_seed(seed)
torch.backends.cudnn.deterministic = True
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)

# numpy RNG
import numpy as np

np.random.seed(seed)

# tf RNG
tf.random.set_seed(seed)
```

### Debugging tests

To start a debugger at the point of the warning, do this:

```bash
pytest tests/utils/test_logging.py -W error::UserWarning --pdb
```

## Working with github actions workflows

To trigger a self-push workflow CI job, you must:

1. Create a new branch on `transformers` origin (not a fork!).
2. The branch name has to start with either `ci_` or `ci-` (`main` triggers it too, but we can't do PRs on
   `main`). It also gets triggered only for specific paths - you can find the up-to-date definition in case it
   changed since this document has been written [here](https://github.com/huggingface/transformers/blob/main/.github/workflows/self-push.yml) under *push:*
3. Create a PR from this branch.
4. Then you can see the job appear [here](https://github.com/huggingface/transformers/actions/workflows/self-push.yml). It may not run right away if there
   is a backlog.




## Testing Experimental CI Features

Testing CI features can be potentially problematic as it can interfere with the normal CI functioning. Therefore if a
new CI feature is to be added, it should be done as following.

1. Create a new dedicated job that tests what needs to be tested
2. The new job must always succeed so that it gives us a green ✓ (details below).
3. Let it run for some days to see that a variety of different PR types get to run on it (user fork branches,
   non-forked branches, branches originating from github.com UI direct file edit, various forced pushes, etc. - there
   are so many) while monitoring the experimental job's logs (not the overall job green as it's purposefully always
   green)
4. When it's clear that everything is solid, then merge the new changes into existing jobs.

That way experiments on CI functionality itself won't interfere with the normal workflow.

Now how can we make the job always succeed while the new CI feature is being developed?

Some CIs, like TravisCI support ignore-step-failure and will report the overall job as successful, but CircleCI and
Github Actions as of this writing don't support that.

So the following workaround can be used:

1. `set +euo pipefail` at the beginning of the run command to suppress most potential failures in the bash script.
2. the last command must be a success: `echo "done"` or just `true` will do

Here is an example:

```yaml
- run:
    name: run CI experiment
    command: |
        set +euo pipefail
        echo "setting run-all-despite-any-errors-mode"
        this_command_will_fail
        echo "but bash continues to run"
        # emulate another failure
        false
        # but the last command must be a success
        echo "during experiment do not remove: reporting success to CI, even if there were failures"
```

For simple commands you could also do:

```bash
cmd_that_may_fail || true
```

Of course, once satisfied with the results, integrate the experimental step or job with the rest of the normal jobs,
while removing `set +euo pipefail` or any other things you may have added to ensure that the experimental job doesn't
interfere with the normal CI functioning.

This whole process would have been much easier if we only could set something like `allow-failure` for the
experimental step, and let it fail without impacting the overall status of PRs. But as mentioned earlier CircleCI and
Github Actions don't support it at the moment.

You can vote for this feature and see where it is at these CI-specific threads:

- [Github Actions:](https://github.com/actions/toolkit/issues/399)
- [CircleCI:](https://ideas.circleci.com/ideas/CCI-I-344)

## DeepSpeed integration

For a PR that involves the DeepSpeed integration, keep in mind our CircleCI PR CI setup doesn't have GPUs. Tests requiring GPUs are run on a different CI nightly. This means if you get a passing CI report in your PR, it doesn’t mean the DeepSpeed tests pass.

To run DeepSpeed tests:

```bash
RUN_SLOW=1 pytest tests/deepspeed/test_deepspeed.py
```

Any changes to the modeling or PyTorch examples code requires running the model zoo tests as well.

```bash
RUN_SLOW=1 pytest tests/deepspeed
```


key: _node_content
string_value:
{"id_": "c3779e30-53c6-4df9-85f7-b1fbda7bee56", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08c2c82b-62e6-4847-bc21-c219691f9ea6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers"}, "hash": "4080bf02bd42c08da311b192ac1c7aa133ecbf6a8a68562c9f70249f67176977", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5aacc5e9-e02e-4d32-9e6c-1dc616a9898f", "node_type": "1", "metadata": {}, "hash": "2630005ea254b3a44b9e379bdcc05906a643079260b0894c7643a80596470c4a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3352, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!---
Copyright 2021 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Performance and Scalability

Training large transformer models and deploying them to production present various challenges.  
During training, the model may require more GPU memory than available or exhibit slow training speed. In the deployment 
phase, the model can struggle to handle the required throughput in a production environment.

This documentation aims to assist you in overcoming these challenges and finding the optimal setting for your use-case. 
The guides are divided into training and inference sections, as each comes with different challenges and solutions. 
Within each section you'll find separate guides for different hardware configurations, such as single GPU vs. multi-GPU 
for training or CPU vs. GPU for inference.

Use this document as your starting point to navigate further to the methods that match your scenario.

## Training

Training large transformer models efficiently requires an accelerator such as a GPU or TPU. The most common case is where 
you have a single GPU. The methods that you can apply to improve training efficiency on a single GPU extend to other setups 
such as multiple GPU. However, there are also techniques that are specific to multi-GPU or CPU training. We cover them in 
separate sections.

* [Methods and tools for efficient training on a single GPU](perf_train_gpu_one): start here to learn common approaches that can help optimize GPU memory utilization, speed up the training, or both. 
* [Multi-GPU training section](perf_train_gpu_many): explore this section to learn about further optimization methods that apply to a multi-GPU settings, such as data, tensor, and pipeline parallelism.
* [CPU training section](perf_train_cpu): learn about mixed precision training on CPU.
* [Efficient Training on Multiple CPUs](perf_train_cpu_many): learn about distributed CPU training.
* [Training on TPU with TensorFlow](perf_train_tpu_tf): if you are new to TPUs, refer to this section for an opinionated introduction to training on TPUs and using XLA. 
* [Custom hardware for training](perf_hardware): find tips and tricks when building your own deep learning rig.
* [Hyperparameter Search using Trainer API](hpo_train)

## Inference

Efficient inference with large models in a production environment can be as challenging as training them. In the following 
sections we go through the steps to run inference on CPU and single/multi-GPU setups.

* [Inference on a single CPU](perf_infer_cpu)
* [Inference on a single GPU](perf_infer_gpu_one)
* [Multi-GPU inference](perf_infer_gpu_one)
* [XLA Integration for TensorFlow Models](tf_xla)


key: _node_content
string_value:
{"id_": "5aacc5e9-e02e-4d32-9e6c-1dc616a9898f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08c2c82b-62e6-4847-bc21-c219691f9ea6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers"}, "hash": "4080bf02bd42c08da311b192ac1c7aa133ecbf6a8a68562c9f70249f67176977", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c3779e30-53c6-4df9-85f7-b1fbda7bee56", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers"}, "hash": "a0b9f20f17aacdbe8c4880837040b0b356989bde3873ad7a6d6fd883483d5b79", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3355, "end_char_idx": 4029, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Training and inference

Here you'll find techniques, tips and tricks that apply whether you are training a model, or running inference with it.

* [Instantiating a big model](big_models)
* [Troubleshooting performance issues](debugging)

## Contribute

This document is far from being complete and a lot more needs to be added, so if you have additions or corrections to 
make please don't hesitate to open a PR or if you aren't sure start an Issue and we can discuss the details there.

When making contributions that A is better than B, please try to include a reproducible benchmark and/or a link to the 
source of that information (unless it comes directly from you).


key: _node_content
string_value:
{"id_": "178342a2-e517-4cd9-8c9f-68bc6c640fa6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5549c1e2-25cb-4734-a807-e25d44dacd6a", "node_type": "1", "metadata": {}, "hash": "86b9274f60676ae660c2b6f1e47b78707b5ff3719e8f22033304d757e3dc5e0b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3580, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# What 🤗 Transformers can do

🤗 Transformers is a library of pretrained state-of-the-art models for natural language processing (NLP), computer vision, and audio and speech processing tasks. Not only does the library contain Transformer models, but it also has non-Transformer models like modern convolutional networks for computer vision tasks. If you look at some of the most popular consumer products today, like smartphones, apps, and televisions, odds are that some kind of deep learning technology is behind it. Want to remove a background object from a picture taken by your smartphone? This is an example of a panoptic segmentation task (don't worry if you don't know what this means yet, we'll describe it in the following sections!). 

This page provides an overview of the different speech and audio, computer vision, and NLP tasks that can be solved with the 🤗 Transformers library in just three lines of code!

## Audio

Audio and speech processing tasks are a little different from the other modalities mainly because audio as an input is a continuous signal. Unlike text, a raw audio waveform can't be neatly split into discrete chunks the way a sentence can be divided into words. To get around this, the raw audio signal is typically sampled at regular intervals. If you take more samples within an interval, the sampling rate is higher, and the audio more closely resembles the original audio source.

Previous approaches preprocessed the audio to extract useful features from it. It is now more common to start audio and speech processing tasks by directly feeding the raw audio waveform to a feature encoder to extract an audio representation. This simplifies the preprocessing step and allows the model to learn the most essential features.

### Audio classification

Audio classification is a task that labels audio data from a predefined set of classes. It is a broad category with many specific applications, some of which include:

* acoustic scene classification: label audio with a scene label ("office", "beach", "stadium")
* acoustic event detection: label audio with a sound event label ("car horn", "whale calling", "glass breaking")
* tagging: label audio containing multiple sounds (birdsongs, speaker identification in a meeting)
* music classification: label music with a genre label ("metal", "hip-hop", "country")

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="audio-classification", model="superb/hubert-base-superb-er")
>>> preds = classifier("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.4532, 'label': 'hap'},
 {'score': 0.3622, 'label': 'sad'},
 {'score': 0.0943, 'label': 'neu'},
 {'score': 0.0903, 'label': 'ang'}]
```

### Automatic speech recognition

Automatic speech recognition (ASR) transcribes speech into text. It is one of the most common audio tasks due partly to speech being such a natural form of human communication. Today, ASR systems are embedded in "smart" technology products like speakers, phones, and cars. We can ask our virtual assistants to play music, set reminders, and tell us the weather. 

But one of the key challenges Transformer architectures have helped with is in low-resource languages. By pretraining on large amounts of speech data, finetuning the model on only one hour of labeled speech data in a low-resource language can still produce high-quality results compared to previous ASR systems trained on 100x more labeled data.


key: _node_content
string_value:
{"id_": "5549c1e2-25cb-4734-a807-e25d44dacd6a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "178342a2-e517-4cd9-8c9f-68bc6c640fa6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "25eb3f029331abd1f689c5175e8884a2c8c5e1defc3ef469088cccf2fd86076a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f11f83ed-dd27-4d3c-968e-12d7f10b2254", "node_type": "1", "metadata": {}, "hash": "4217f64a15a9898c2b855905d6c1a845687f16da3f284c4416b095abb8789ae4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2936, "end_char_idx": 6277, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It is one of the most common audio tasks due partly to speech being such a natural form of human communication. Today, ASR systems are embedded in "smart" technology products like speakers, phones, and cars. We can ask our virtual assistants to play music, set reminders, and tell us the weather. 

But one of the key challenges Transformer architectures have helped with is in low-resource languages. By pretraining on large amounts of speech data, finetuning the model on only one hour of labeled speech data in a low-resource language can still produce high-quality results compared to previous ASR systems trained on 100x more labeled data.

```py
>>> from transformers import pipeline

>>> transcriber = pipeline(task="automatic-speech-recognition", model="openai/whisper-small")
>>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
{'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'}
```

## Computer vision

One of the first and earliest successful computer vision tasks was recognizing images of zip code numbers using a [convolutional neural network (CNN)](glossary#convolution). An image is composed of pixels, and each pixel has a numerical value. This makes it easy to represent an image as a matrix of pixel values. Each particular combination of pixel values describes the colors of an image. 

Two general ways computer vision tasks can be solved are:

1. Use convolutions to learn the hierarchical features of an image from low-level features to high-level abstract things.
2. Split an image into patches and use a Transformer to gradually learn how each image patch is related to each other to form an image. Unlike the bottom-up approach favored by a CNN, this is kind of like starting out with a blurry image and then gradually bringing it into focus.

### Image classification

Image classification labels an entire image from a predefined set of classes. Like most classification tasks, there are many practical use cases for image classification, some of which include:

* healthcare: label medical images to detect disease or monitor patient health
* environment: label satellite images to monitor deforestation, inform wildland management or detect wildfires
* agriculture: label images of crops to monitor plant health or satellite images for land use monitoring 
* ecology: label images of animal or plant species to monitor wildlife populations or track endangered species

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="image-classification")
>>> preds = classifier(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.4335, 'label': 'lynx, catamount'}
{'score': 0.0348, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}
{'score': 0.0324, 'label': 'snow leopard, ounce, Panthera uncia'}
{'score': 0.0239, 'label': 'Egyptian cat'}
{'score': 0.0229, 'label': 'tiger cat'}
```

### Object detection

Unlike image classification, object detection identifies multiple objects within an image and the objects' positions in an image (defined by the bounding box).


key: _node_content
string_value:
{"id_": "f11f83ed-dd27-4d3c-968e-12d7f10b2254", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5549c1e2-25cb-4734-a807-e25d44dacd6a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "e3b26371b7397c4bed0c6457ab0a936cf202e2e15addb338512e81f6f599633b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "edcfbf98-d337-4a3b-b09b-aa2d67cae413", "node_type": "1", "metadata": {}, "hash": "cb7651ddc995abe45051847dfddef6855e5f108001faa70a06f9a62d6d23115e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4524, "end_char_idx": 8045, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Split an image into patches and use a Transformer to gradually learn how each image patch is related to each other to form an image. Unlike the bottom-up approach favored by a CNN, this is kind of like starting out with a blurry image and then gradually bringing it into focus.

### Image classification

Image classification labels an entire image from a predefined set of classes. Like most classification tasks, there are many practical use cases for image classification, some of which include:

* healthcare: label medical images to detect disease or monitor patient health
* environment: label satellite images to monitor deforestation, inform wildland management or detect wildfires
* agriculture: label images of crops to monitor plant health or satellite images for land use monitoring 
* ecology: label images of animal or plant species to monitor wildlife populations or track endangered species

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="image-classification")
>>> preds = classifier(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.4335, 'label': 'lynx, catamount'}
{'score': 0.0348, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}
{'score': 0.0324, 'label': 'snow leopard, ounce, Panthera uncia'}
{'score': 0.0239, 'label': 'Egyptian cat'}
{'score': 0.0229, 'label': 'tiger cat'}
```

### Object detection

Unlike image classification, object detection identifies multiple objects within an image and the objects' positions in an image (defined by the bounding box). Some example applications of object detection include:

* self-driving vehicles: detect everyday traffic objects such as other vehicles, pedestrians, and traffic lights
* remote sensing: disaster monitoring, urban planning, and weather forecasting
* defect detection: detect cracks or structural damage in buildings, and manufacturing defects

```py
>>> from transformers import pipeline

>>> detector = pipeline(task="object-detection")
>>> preds = detector(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"], "box": pred["box"]} for pred in preds]
>>> preds
[{'score': 0.9865,
  'label': 'cat',
  'box': {'xmin': 178, 'ymin': 154, 'xmax': 882, 'ymax': 598}}]
```

### Image segmentation

Image segmentation is a pixel-level task that assigns every pixel in an image to a class. It differs from object detection, which uses bounding boxes to label and predict objects in an image because segmentation is more granular. Segmentation can detect objects at a pixel-level. There are several types of image segmentation:

* instance segmentation: in addition to labeling the class of an object, it also labels each distinct instance of an object ("dog-1", "dog-2")
* panoptic segmentation: a combination of semantic and instance segmentation; it labels each pixel with a semantic class **and** each distinct instance of an object

Segmentation tasks are helpful in self-driving vehicles to create a pixel-level map of the world around them so they can navigate safely around pedestrians and other vehicles. It is also useful for medical imaging, where the task's finer granularity can help identify abnormal cells or organ features.


key: _node_content
string_value:
{"id_": "edcfbf98-d337-4a3b-b09b-aa2d67cae413", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f11f83ed-dd27-4d3c-968e-12d7f10b2254", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "54a2f4ef00d609ba4184951458019bc5c3d5d1e549af25fe7046e5ec03c6c544", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5dd5e2d7-e10f-41bc-9e96-f2ecd9ace3f5", "node_type": "1", "metadata": {}, "hash": "ee65d4d906f42f271bce8e33034bbd9f2632a73029434b0e9788e112ba4c8957", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6278, "end_char_idx": 9846, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Some example applications of object detection include:

* self-driving vehicles: detect everyday traffic objects such as other vehicles, pedestrians, and traffic lights
* remote sensing: disaster monitoring, urban planning, and weather forecasting
* defect detection: detect cracks or structural damage in buildings, and manufacturing defects

```py
>>> from transformers import pipeline

>>> detector = pipeline(task="object-detection")
>>> preds = detector(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"], "box": pred["box"]} for pred in preds]
>>> preds
[{'score': 0.9865,
  'label': 'cat',
  'box': {'xmin': 178, 'ymin': 154, 'xmax': 882, 'ymax': 598}}]
```

### Image segmentation

Image segmentation is a pixel-level task that assigns every pixel in an image to a class. It differs from object detection, which uses bounding boxes to label and predict objects in an image because segmentation is more granular. Segmentation can detect objects at a pixel-level. There are several types of image segmentation:

* instance segmentation: in addition to labeling the class of an object, it also labels each distinct instance of an object ("dog-1", "dog-2")
* panoptic segmentation: a combination of semantic and instance segmentation; it labels each pixel with a semantic class **and** each distinct instance of an object

Segmentation tasks are helpful in self-driving vehicles to create a pixel-level map of the world around them so they can navigate safely around pedestrians and other vehicles. It is also useful for medical imaging, where the task's finer granularity can help identify abnormal cells or organ features. Image segmentation can also be used in ecommerce to virtually try on clothes or create augmented reality experiences by overlaying objects in the real world through your camera.

```py
>>> from transformers import pipeline

>>> segmenter = pipeline(task="image-segmentation")
>>> preds = segmenter(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.9879, 'label': 'LABEL_184'}
{'score': 0.9973, 'label': 'snow'}
{'score': 0.9972, 'label': 'cat'}
```

### Depth estimation

Depth estimation predicts the distance of each pixel in an image from the camera. This computer vision task is especially important for scene understanding and reconstruction. For example, in self-driving cars, vehicles need to understand how far objects like pedestrians, traffic signs, and other vehicles are to avoid obstacles and collisions. Depth information is also helpful for constructing 3D representations from 2D images and can be used to create high-quality 3D representations of biological structures or buildings.

There are two approaches to depth estimation:

* stereo: depths are estimated by comparing two images of the same image from slightly different angles
* monocular: depths are estimated from a single image

```py
>>> from transformers import pipeline

>>> depth_estimator = pipeline(task="depth-estimation")
>>> preds = depth_estimator(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
```

## Natural language processing

NLP tasks are among the most common types of tasks because text is such a natural way for us to communicate.


key: _node_content
string_value:
{"id_": "5dd5e2d7-e10f-41bc-9e96-f2ecd9ace3f5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "edcfbf98-d337-4a3b-b09b-aa2d67cae413", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "bf41c7dba5e3caa636cd4f5c375c68a0d98dc973c60b19ca9337fa1bf6a385a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "18fbce96-e197-4062-984b-c7bcd8e6a583", "node_type": "1", "metadata": {}, "hash": "f01aa8c49f0bbe17fa282f2a40693c8394b6bf925081bc63da7356def6f64d05", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8046, "end_char_idx": 11630, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Image segmentation can also be used in ecommerce to virtually try on clothes or create augmented reality experiences by overlaying objects in the real world through your camera.

```py
>>> from transformers import pipeline

>>> segmenter = pipeline(task="image-segmentation")
>>> preds = segmenter(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.9879, 'label': 'LABEL_184'}
{'score': 0.9973, 'label': 'snow'}
{'score': 0.9972, 'label': 'cat'}
```

### Depth estimation

Depth estimation predicts the distance of each pixel in an image from the camera. This computer vision task is especially important for scene understanding and reconstruction. For example, in self-driving cars, vehicles need to understand how far objects like pedestrians, traffic signs, and other vehicles are to avoid obstacles and collisions. Depth information is also helpful for constructing 3D representations from 2D images and can be used to create high-quality 3D representations of biological structures or buildings.

There are two approaches to depth estimation:

* stereo: depths are estimated by comparing two images of the same image from slightly different angles
* monocular: depths are estimated from a single image

```py
>>> from transformers import pipeline

>>> depth_estimator = pipeline(task="depth-estimation")
>>> preds = depth_estimator(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
```

## Natural language processing

NLP tasks are among the most common types of tasks because text is such a natural way for us to communicate. To get text into a format recognized by a model, it needs to be tokenized. This means dividing a sequence of text into separate words or subwords (tokens) and then converting these tokens into numbers. As a result, you can represent a sequence of text as a sequence of numbers, and once you have a sequence of numbers, it can be input into a model to solve all sorts of NLP tasks!

### Text classification

Like classification tasks in any modality, text classification labels a sequence of text (it can be sentence-level, a paragraph, or a document) from a predefined set of classes. There are many practical applications for text classification, some of which include:

* sentiment analysis: label text according to some polarity like `positive` or `negative` which can inform and support decision-making in fields like politics, finance, and marketing
* content classification: label text according to some topic to help organize and filter information in news and social media feeds (`weather`, `sports`, `finance`, etc.)

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="sentiment-analysis")
>>> preds = classifier("Hugging Face is the best thing since sliced bread!")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.9991, 'label': 'POSITIVE'}]
```

### Token classification

In any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as [tokens](glossary#token). Token classification assigns each token a label from a predefined set of classes. 

Two common types of token classification are:

* named entity recognition (NER): label a token according to an entity category like organization, person, location or date.


key: _node_content
string_value:
{"id_": "18fbce96-e197-4062-984b-c7bcd8e6a583", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5dd5e2d7-e10f-41bc-9e96-f2ecd9ace3f5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "527efa3019b5c9c54efa55459975989184db1f2eea3ae00592a690b29705566e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c29f73b2-dc47-4d6a-a462-fbddc81d3ee9", "node_type": "1", "metadata": {}, "hash": "dacb7c0bad5c895b8c6d63b6947868cef3eb2e897f90ce9e2402c4d66d752648", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9847, "end_char_idx": 12157, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To get text into a format recognized by a model, it needs to be tokenized. This means dividing a sequence of text into separate words or subwords (tokens) and then converting these tokens into numbers. As a result, you can represent a sequence of text as a sequence of numbers, and once you have a sequence of numbers, it can be input into a model to solve all sorts of NLP tasks!

### Text classification

Like classification tasks in any modality, text classification labels a sequence of text (it can be sentence-level, a paragraph, or a document) from a predefined set of classes. There are many practical applications for text classification, some of which include:

* sentiment analysis: label text according to some polarity like `positive` or `negative` which can inform and support decision-making in fields like politics, finance, and marketing
* content classification: label text according to some topic to help organize and filter information in news and social media feeds (`weather`, `sports`, `finance`, etc.)

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="sentiment-analysis")
>>> preds = classifier("Hugging Face is the best thing since sliced bread!")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.9991, 'label': 'POSITIVE'}]
```

### Token classification

In any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as [tokens](glossary#token). Token classification assigns each token a label from a predefined set of classes. 

Two common types of token classification are:

* named entity recognition (NER): label a token according to an entity category like organization, person, location or date. NER is especially popular in biomedical settings, where it can label genes, proteins, and drug names.
* part-of-speech tagging (POS): label a token according to its part-of-speech like noun, verb, or adjective. POS is useful for helping translation systems understand how two identical words are grammatically different (bank as a noun versus bank as a verb).

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="ner")
>>> preds = classifier("Hugging Face is a French company based in New York City.")


key: _node_content
string_value:
{"id_": "c29f73b2-dc47-4d6a-a462-fbddc81d3ee9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "18fbce96-e197-4062-984b-c7bcd8e6a583", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "970d1e6e4dbda2b42cb19bfa0f039b5b6aefa50d500204ddba357a0ba50e0380", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "460c5ee5-3190-475f-9762-a00e62d6a5d5", "node_type": "1", "metadata": {}, "hash": "f29c07dad20ae21722adf83d6536ae294fb99f95b75a4a1d9e83061eaf2b83ec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10432, "end_char_idx": 13296, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
There are many practical applications for text classification, some of which include:

* sentiment analysis: label text according to some polarity like `positive` or `negative` which can inform and support decision-making in fields like politics, finance, and marketing
* content classification: label text according to some topic to help organize and filter information in news and social media feeds (`weather`, `sports`, `finance`, etc.)

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="sentiment-analysis")
>>> preds = classifier("Hugging Face is the best thing since sliced bread!")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.9991, 'label': 'POSITIVE'}]
```

### Token classification

In any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as [tokens](glossary#token). Token classification assigns each token a label from a predefined set of classes. 

Two common types of token classification are:

* named entity recognition (NER): label a token according to an entity category like organization, person, location or date. NER is especially popular in biomedical settings, where it can label genes, proteins, and drug names.
* part-of-speech tagging (POS): label a token according to its part-of-speech like noun, verb, or adjective. POS is useful for helping translation systems understand how two identical words are grammatically different (bank as a noun versus bank as a verb).

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="ner")
>>> preds = classifier("Hugging Face is a French company based in New York City.")
>>> preds = [
...     {
...         "entity": pred["entity"],
...         "score": round(pred["score"], 4),
...         "index": pred["index"],
...         "word": pred["word"],
...         "start": pred["start"],
...         "end": pred["end"],
...     }
...     for pred in preds
... ]
>>> print(*preds, sep="\n")
{'entity': 'I-ORG', 'score': 0.9968, 'index': 1, 'word': 'Hu', 'start': 0, 'end': 2}
{'entity': 'I-ORG', 'score': 0.9293, 'index': 2, 'word': '##gging', 'start': 2, 'end': 7}
{'entity': 'I-ORG', 'score': 0.9763, 'index': 3, 'word': 'Face', 'start': 8, 'end': 12}
{'entity': 'I-MISC', 'score': 0.9983, 'index': 6, 'word': 'French', 'start': 18, 'end': 24}
{'entity': 'I-LOC', 'score': 0.999, 'index': 10, 'word': 'New', 'start': 42, 'end': 45}
{'entity': 'I-LOC', 'score': 0.9987, 'index': 11, 'word': 'York', 'start': 46, 'end': 50}
{'entity': 'I-LOC', 'score': 0.9992, 'index': 12, 'word': 'City', 'start': 51, 'end': 55}
```

### Question answering

Question answering is another token-level task that returns an answer to a question, sometimes with context (open-domain) and other times without context (closed-domain).


key: _node_content
string_value:
{"id_": "460c5ee5-3190-475f-9762-a00e62d6a5d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c29f73b2-dc47-4d6a-a462-fbddc81d3ee9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "662a057f46d7e3476debb5ddf37a77744730eaa129df10537f27353be64c6bbc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a8cb3f4a-39c4-4bdc-82d2-ae62888686d9", "node_type": "1", "metadata": {}, "hash": "3e794fd959c9b63e7024ab64f859bce441a06f7db3ea2086fe79372522f0ecde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13297, "end_char_idx": 16982, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This task happens whenever we ask a virtual assistant something like whether a restaurant is open. It can also provide customer or technical support and help search engines retrieve the relevant information you're asking for. 

There are two common types of question answering:

* extractive: given a question and some context, the answer is a span of text from the context the model must extract
* abstractive: given a question and some context, the answer is generated from the context; this approach is handled by the [`Text2TextGenerationPipeline`] instead of the [`QuestionAnsweringPipeline`] shown below


```py
>>> from transformers import pipeline

>>> question_answerer = pipeline(task="question-answering")
>>> preds = question_answerer(
...     question="What is the name of the repository?",
...     context="The name of the repository is huggingface/transformers",
... )
>>> print(
...     f"score: {round(preds['score'], 4)}, start: {preds['start']}, end: {preds['end']}, answer: {preds['answer']}"
... )
score: 0.9327, start: 30, end: 54, answer: huggingface/transformers
```

### Summarization

Summarization creates a shorter version of a text from a longer one while trying to preserve most of the meaning of the original document. Summarization is a sequence-to-sequence task; it outputs a shorter text sequence than the input. There are a lot of long-form documents that can be summarized to help readers quickly understand the main points. Legislative bills, legal and financial documents, patents, and scientific papers are a few examples of documents that could be summarized to save readers time and serve as a reading aid.

Like question answering, there are two types of summarization:

* extractive: identify and extract the most important sentences from the original text
* abstractive: generate the target summary (which may include new words not in the input document) from the original text; the [`SummarizationPipeline`] uses the abstractive approach

```py
>>> from transformers import pipeline

>>> summarizer = pipeline(task="summarization")
>>> summarizer(
...     "In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles."
... )
[{'summary_text': ' The Transformer is the first sequence transduction model based entirely on attention . It replaces the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention . For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers .'}]
```

### Translation

Translation converts a sequence of text in one language to another. It is important in helping people from different backgrounds communicate with each other, help translate content to reach wider audiences, and even be a learning tool to help people learn a new language. Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence. 

In the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages.


key: _node_content
string_value:
{"id_": "a8cb3f4a-39c4-4bdc-82d2-ae62888686d9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "460c5ee5-3190-475f-9762-a00e62d6a5d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "a00de75ac4f3512ce364808aca0bcdc175b5652b26c21d546348ac9cd30e67fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e9feb5cb-f3e1-4107-bb29-7a4090f8646c", "node_type": "1", "metadata": {}, "hash": "7d418c436298dbea029ff6a7f74cd864b74efeb32e4b425332107fab58fdc26c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14946, "end_char_idx": 18535, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Like question answering, there are two types of summarization:

* extractive: identify and extract the most important sentences from the original text
* abstractive: generate the target summary (which may include new words not in the input document) from the original text; the [`SummarizationPipeline`] uses the abstractive approach

```py
>>> from transformers import pipeline

>>> summarizer = pipeline(task="summarization")
>>> summarizer(
...     "In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles."
... )
[{'summary_text': ' The Transformer is the first sequence transduction model based entirely on attention . It replaces the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention . For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers .'}]
```

### Translation

Translation converts a sequence of text in one language to another. It is important in helping people from different backgrounds communicate with each other, help translate content to reach wider audiences, and even be a learning tool to help people learn a new language. Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence. 

In the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages.

```py
>>> from transformers import pipeline

>>> text = "translate English to French: Hugging Face is a community-based open-source platform for machine learning."
>>> translator = pipeline(task="translation", model="google-t5/t5-small")
>>> translator(text)
[{'translation_text': "Hugging Face est une tribune communautaire de l'apprentissage des machines."}]
```

### Language modeling

Language modeling is a task that predicts a word in a sequence of text. It has become a very popular NLP task because a pretrained language model can be finetuned for many other downstream tasks. Lately, there has been a lot of interest in large language models (LLMs) which demonstrate zero- or few-shot learning. This means the model can solve tasks it wasn't explicitly trained to do! Language models can be used to generate fluent and convincing text, though you need to be careful since the text may not always be accurate.

There are two types of language modeling:

* causal: the model's objective is to predict the next token in a sequence, and future tokens are masked

    ```py
    >>> from transformers import pipeline

    >>> prompt = "Hugging Face is a community-based open-source platform for machine learning."
    >>> generator = pipeline(task="text-generation")
    >>> generator(prompt)  # doctest: +SKIP
    ```

* masked: the model's objective is to predict a masked token in a sequence with full access to the tokens in the sequence
    
    ```py
    >>> text = "Hugging Face is a community-based open-source <mask> for machine learning."


key: _node_content
string_value:
{"id_": "e9feb5cb-f3e1-4107-bb29-7a4090f8646c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a8cb3f4a-39c4-4bdc-82d2-ae62888686d9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "35c4896e47ed93ab7b068296a9a40139f63c80acb81a26879fe122c849d93b1c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ca96c93d-498f-4303-bc16-5fbc0467d4ac", "node_type": "1", "metadata": {}, "hash": "0da49853a00474ed338686b9ccff22384b3dcbe7afb95a3a197d607cc3c1f83b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16641, "end_char_idx": 20295, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence. 

In the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages.

```py
>>> from transformers import pipeline

>>> text = "translate English to French: Hugging Face is a community-based open-source platform for machine learning."
>>> translator = pipeline(task="translation", model="google-t5/t5-small")
>>> translator(text)
[{'translation_text': "Hugging Face est une tribune communautaire de l'apprentissage des machines."}]
```

### Language modeling

Language modeling is a task that predicts a word in a sequence of text. It has become a very popular NLP task because a pretrained language model can be finetuned for many other downstream tasks. Lately, there has been a lot of interest in large language models (LLMs) which demonstrate zero- or few-shot learning. This means the model can solve tasks it wasn't explicitly trained to do! Language models can be used to generate fluent and convincing text, though you need to be careful since the text may not always be accurate.

There are two types of language modeling:

* causal: the model's objective is to predict the next token in a sequence, and future tokens are masked

    ```py
    >>> from transformers import pipeline

    >>> prompt = "Hugging Face is a community-based open-source platform for machine learning."
    >>> generator = pipeline(task="text-generation")
    >>> generator(prompt)  # doctest: +SKIP
    ```

* masked: the model's objective is to predict a masked token in a sequence with full access to the tokens in the sequence
    
    ```py
    >>> text = "Hugging Face is a community-based open-source <mask> for machine learning."
    >>> fill_mask = pipeline(task="fill-mask")
    >>> preds = fill_mask(text, top_k=1)
    >>> preds = [
    ...     {
    ...         "score": round(pred["score"], 4),
    ...         "token": pred["token"],
    ...         "token_str": pred["token_str"],
    ...         "sequence": pred["sequence"],
    ...     }
    ...     for pred in preds
    ... ]
    >>> preds
    [{'score': 0.2236,
      'token': 1761,
      'token_str': ' platform',
      'sequence': 'Hugging Face is a community-based open-source platform for machine learning.'}]
    ```

## Multimodal

Multimodal tasks require a model to process multiple data modalities (text, image, audio, video) to solve a particular problem. Image captioning is an example of a multimodal task where the model takes an image as input and outputs a sequence of text describing the image or some properties of the image. 

Although multimodal models work with different data types or modalities, internally, the preprocessing steps help the model convert all the data types into embeddings (vectors or list of numbers that holds meaningful information about the data). For a task like image captioning, the model learns relationships between image embeddings and text embeddings.

### Document question answering

Document question answering is a task that answers natural language questions from a document. Unlike a token-level question answering task which takes text as input, document question answering takes an image of a document as input along with a question about the document and returns an answer. Document question answering can be used to parse structured documents and extract key information from it. In the example below, the total amount and change due can be extracted from a receipt.


key: _node_content
string_value:
{"id_": "ca96c93d-498f-4303-bc16-5fbc0467d4ac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9e2b4755-acad-46e3-b658-4cb47bb3f994", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9feb5cb-f3e1-4107-bb29-7a4090f8646c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "54084d551ddb6e20e9b0c3b76e7deb11d59952c86b4cdf7ba8c5c1bf7ba8e4a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18540, "end_char_idx": 21111, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> fill_mask = pipeline(task="fill-mask")
    >>> preds = fill_mask(text, top_k=1)
    >>> preds = [
    ...     {
    ...         "score": round(pred["score"], 4),
    ...         "token": pred["token"],
    ...         "token_str": pred["token_str"],
    ...         "sequence": pred["sequence"],
    ...     }
    ...     for pred in preds
    ... ]
    >>> preds
    [{'score': 0.2236,
      'token': 1761,
      'token_str': ' platform',
      'sequence': 'Hugging Face is a community-based open-source platform for machine learning.'}]
    ```

## Multimodal

Multimodal tasks require a model to process multiple data modalities (text, image, audio, video) to solve a particular problem. Image captioning is an example of a multimodal task where the model takes an image as input and outputs a sequence of text describing the image or some properties of the image. 

Although multimodal models work with different data types or modalities, internally, the preprocessing steps help the model convert all the data types into embeddings (vectors or list of numbers that holds meaningful information about the data). For a task like image captioning, the model learns relationships between image embeddings and text embeddings.

### Document question answering

Document question answering is a task that answers natural language questions from a document. Unlike a token-level question answering task which takes text as input, document question answering takes an image of a document as input along with a question about the document and returns an answer. Document question answering can be used to parse structured documents and extract key information from it. In the example below, the total amount and change due can be extracted from a receipt.

```py
>>> from transformers import pipeline
>>> from PIL import Image
>>> import requests

>>> url = "https://huggingface.co/datasets/hf-internal-testing/example-documents/resolve/main/jpeg_images/2.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> doc_question_answerer = pipeline("document-question-answering", model="magorshunov/layoutlm-invoices")
>>> preds = doc_question_answerer(
...     question="What is the total amount?",
...     image=image,
... )
>>> preds
[{'score': 0.8531, 'answer': '17,000', 'start': 4, 'end': 4}]
```

Hopefully, this page has given you some more background information about all the types of tasks in each modality and the practical importance of each one. In the next [section](tasks_explained), you'll learn **how** 🤗 Transformers work to solve these tasks.


key: _node_content
string_value:
{"id_": "8cef2e79-f19e-4980-8a73-4cecb9e177e5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4d321a14-ef85-42ba-acf2-421c1a58b1a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "7d797771b1f9101ed3be1753409bd31a324784969f01bfec031a47be93a6c7d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f24bbbaf-daa3-4967-aed7-97f4a45f8c02", "node_type": "1", "metadata": {}, "hash": "017ffcb932fff7f132a0bcacc761fc07a8eedf1027a728bf51701e62a23b5d00", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2284, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Hyperparameter Search using Trainer API

🤗 Transformers provides a [`Trainer`] class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The [`Trainer`] provides API for hyperparameter search. This doc shows how to enable it in example. 

## Hyperparameter Search backend

[`Trainer`] supports four hyperparameter search backends currently:
[optuna](https://optuna.org/), [sigopt](https://sigopt.com/), [raytune](https://docs.ray.io/en/latest/tune/index.html) and [wandb](https://wandb.ai/site/sweeps).

you should install them before using them as the hyperparameter search backend
```bash
pip install optuna/sigopt/wandb/ray[tune] 
```

## How to enable Hyperparameter search in example

Define the hyperparameter search space, different backends need different format.

For sigopt, see sigopt [object_parameter](https://docs.sigopt.com/ai-module-api-references/api_reference/objects/object_parameter), it's like following:
```py
>>> def sigopt_hp_space(trial):
...     return [
...         {"bounds": {"min": 1e-6, "max": 1e-4}, "name": "learning_rate", "type": "double"},
...         {
...             "categorical_values": ["16", "32", "64", "128"],
...             "name": "per_device_train_batch_size",
...             "type": "categorical",
...         },
...     ]
```

For optuna, see optuna [object_parameter](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html#sphx-glr-tutorial-10-key-features-002-configurations-py), it's like following:

```py
>>> def optuna_hp_space(trial):
...     return {
...         "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
...         "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [16, 32, 64, 128]),
...     }
```

Optuna provides multi-objective HPO. You can pass `direction` in `hyperparameter_search` and define your own compute_objective to return multiple objective values. The Pareto Front (`List[BestRun]`) will be returned in hyperparameter_search, you should refer to the test case `TrainerHyperParameterMultiObjectOptunaIntegrationTest` in [test_trainer](https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py).


key: _node_content
string_value:
{"id_": "f24bbbaf-daa3-4967-aed7-97f4a45f8c02", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4d321a14-ef85-42ba-acf2-421c1a58b1a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "7d797771b1f9101ed3be1753409bd31a324784969f01bfec031a47be93a6c7d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8cef2e79-f19e-4980-8a73-4cecb9e177e5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "bed946cff9e56eca9bd01f48b6c4e14935230cf29fae29f715f4721e6c0ab747", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4a5710a-37af-4b02-a172-4ab4d80e3bcc", "node_type": "1", "metadata": {}, "hash": "8145890f6c49bce81fe9d1fb61705892269fbf1b8174c3093ed66350e7eead1a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 853, "end_char_idx": 4396, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For sigopt, see sigopt [object_parameter](https://docs.sigopt.com/ai-module-api-references/api_reference/objects/object_parameter), it's like following:
```py
>>> def sigopt_hp_space(trial):
...     return [
...         {"bounds": {"min": 1e-6, "max": 1e-4}, "name": "learning_rate", "type": "double"},
...         {
...             "categorical_values": ["16", "32", "64", "128"],
...             "name": "per_device_train_batch_size",
...             "type": "categorical",
...         },
...     ]
```

For optuna, see optuna [object_parameter](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html#sphx-glr-tutorial-10-key-features-002-configurations-py), it's like following:

```py
>>> def optuna_hp_space(trial):
...     return {
...         "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
...         "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [16, 32, 64, 128]),
...     }
```

Optuna provides multi-objective HPO. You can pass `direction` in `hyperparameter_search` and define your own compute_objective to return multiple objective values. The Pareto Front (`List[BestRun]`) will be returned in hyperparameter_search, you should refer to the test case `TrainerHyperParameterMultiObjectOptunaIntegrationTest` in [test_trainer](https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py). It's like following

```py
>>> best_trials = trainer.hyperparameter_search(
...     direction=["minimize", "maximize"],
...     backend="optuna",
...     hp_space=optuna_hp_space,
...     n_trials=20,
...     compute_objective=compute_objective,
... )
```

For raytune, see raytune [object_parameter](https://docs.ray.io/en/latest/tune/api/search_space.html), it's like following:

```py
>>> def ray_hp_space(trial):
...     return {
...         "learning_rate": tune.loguniform(1e-6, 1e-4),
...         "per_device_train_batch_size": tune.choice([16, 32, 64, 128]),
...     }
```

For wandb, see wandb [object_parameter](https://docs.wandb.ai/guides/sweeps/configuration), it's like following:

```py
>>> def wandb_hp_space(trial):
...     return {
...         "method": "random",
...         "metric": {"name": "objective", "goal": "minimize"},
...         "parameters": {
...             "learning_rate": {"distribution": "uniform", "min": 1e-6, "max": 1e-4},
...             "per_device_train_batch_size": {"values": [16, 32, 64, 128]},
...         },
...     }
```

Define a `model_init` function and pass it to the [`Trainer`], as an example:
```py
>>> def model_init(trial):
...     return AutoModelForSequenceClassification.from_pretrained(
...         model_args.model_name_or_path,
...         from_tf=bool(".ckpt" in model_args.model_name_or_path),
...         config=config,
...         cache_dir=model_args.cache_dir,
...         revision=model_args.model_revision,
...         token=True if model_args.use_auth_token else None,
...     )
```

Create a [`Trainer`] with your `model_init` function, training arguments, training and test datasets, and evaluation function:

```py
>>> trainer = Trainer(
...     model=None,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
...     tokenizer=tokenizer,
...     model_init=model_init,
...     data_collator=data_collator,
... )
```

Call hyperparameter search, get the best trial parameters, backend could be `"optuna"`/`"sigopt"`/`"wandb"`/`"ray"`.


key: _node_content
string_value:
{"id_": "e4a5710a-37af-4b02-a172-4ab4d80e3bcc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4d321a14-ef85-42ba-acf2-421c1a58b1a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "7d797771b1f9101ed3be1753409bd31a324784969f01bfec031a47be93a6c7d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f24bbbaf-daa3-4967-aed7-97f4a45f8c02", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "fd4ac6bd272e6f7923fb7bc19e055aa2ed159d27d69db3cff828897d8f462865", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4397, "end_char_idx": 5125, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
direction can be`"minimize"` or `"maximize"`, which indicates whether to optimize greater or lower objective.

You could define your own compute_objective function, if not defined, the default compute_objective will be called, and the sum of eval metric like f1 is returned as objective value.

```py
>>> best_trial = trainer.hyperparameter_search(
...     direction="maximize",
...     backend="optuna",
...     hp_space=optuna_hp_space,
...     n_trials=20,
...     compute_objective=compute_objective,
... )
```

## Hyperparameter search For DDP finetune
Currently, Hyperparameter search for DDP is enabled for optuna and sigopt. Only the rank-zero process will generate the search trial and pass the argument to other ranks.


key: _node_content
string_value:
{"id_": "72a28861-03c5-49d1-99e4-f67f55983fcd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "47679084-01a3-4a08-9315-522e923de21a", "node_type": "1", "metadata": {}, "hash": "005e41af40faf253ff851d022474fb9fc54a4ed3782803c354845a565c76f040", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Efficient Training on Multiple GPUs

If training a model on a single GPU is too slow or if the model's weights do not fit in a single GPU's memory, transitioning 
to a multi-GPU setup may be a viable option. Prior to making this transition, thoroughly explore all the strategies covered 
in the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one) as they are universally applicable 
to model training on any number of GPUs. Once you have employed those strategies and found them insufficient for your 
case on a single GPU, consider moving to multiple GPUs.

Transitioning from a single GPU to multiple GPUs requires the introduction of some form of parallelism, as the workload 
must be distributed across the resources. Multiple techniques can be employed to achieve parallelism, such as data 
parallelism, tensor parallelism, and pipeline parallelism. It's important to note that there isn't a one-size-fits-all 
solution, and the optimal settings depend on the specific hardware configuration you are using. 

This guide offers an in-depth overview of individual types of parallelism, as well as guidance on ways to combine   
techniques and choosing an appropriate approach. For step-by-step tutorials on distributed training, please refer to
the [🤗 Accelerate documentation](https://huggingface.co/docs/accelerate/index). 

<Tip>

While the main concepts discussed in this guide are likely applicable across frameworks, here we focus on 
PyTorch-based implementations.

</Tip>

Before diving deeper into the specifics of each technique, let's go over the rough decision process when training 
large models on a large infrastructure.

## Scalability strategy

Begin by estimating how much vRAM is required to train your model. For models hosted on the 🤗 Hub, use our 
[Model Memory Calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage), which gives you 
accurate calculations within a few percent margin.  

**Parallelization strategy for a single Node / multi-GPU setup**

When training a model on a single node with multiple GPUs, your choice of parallelization strategy can significantly 
impact performance. Here's a breakdown of your options:

**Case 1: Your model fits onto a single GPU**

If your model can comfortably fit onto a single GPU, you have two primary options:

1. DDP - Distributed DataParallel
2. [Zero Redundancy Optimizer (ZeRO)](https://arxiv.org/abs/1910.02054) - depending on the situation and configuration used, this method may or may not be faster, however, it's worth experimenting with it.

**Case 2: Your model doesn't fit onto a single GPU:**

If your model is too large for a single GPU, you have several alternatives to consider:

1. PipelineParallel (PP)
2. [ZeRO](https://arxiv.org/abs/1910.02054)
3. [TensorParallel](#tensor-parallelism) (TP)

With very fast inter-node connectivity (e.g., NVLINK or NVSwitch) all three strategies (PP, ZeRO, TP) should result in 
similar performance. However, without these, PP will be faster than TP or ZeRO. The degree of TP may also 
make a difference. It's best to experiment with your specific setup to determine the most suitable strategy.

TP is almost always used within a single node. That is TP size <= GPUs per node.

**Case 3: Largest layer of your model does not fit onto a single GPU**

1.


key: _node_content
string_value:
{"id_": "47679084-01a3-4a08-9315-522e923de21a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "72a28861-03c5-49d1-99e4-f67f55983fcd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "edda93c2c16079dafb7ff6b27bd0bcfd8e4265d353887ce53172e1b2412a8cdd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9daab811-bac6-4350-b397-a99814b4d21d", "node_type": "1", "metadata": {}, "hash": "b48e0432309b056879d994ae6b0d5a17be52371a5d7e868384d54788ca932a74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1761, "end_char_idx": 4875, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For models hosted on the 🤗 Hub, use our 
[Model Memory Calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage), which gives you 
accurate calculations within a few percent margin.  

**Parallelization strategy for a single Node / multi-GPU setup**

When training a model on a single node with multiple GPUs, your choice of parallelization strategy can significantly 
impact performance. Here's a breakdown of your options:

**Case 1: Your model fits onto a single GPU**

If your model can comfortably fit onto a single GPU, you have two primary options:

1. DDP - Distributed DataParallel
2. [Zero Redundancy Optimizer (ZeRO)](https://arxiv.org/abs/1910.02054) - depending on the situation and configuration used, this method may or may not be faster, however, it's worth experimenting with it.

**Case 2: Your model doesn't fit onto a single GPU:**

If your model is too large for a single GPU, you have several alternatives to consider:

1. PipelineParallel (PP)
2. [ZeRO](https://arxiv.org/abs/1910.02054)
3. [TensorParallel](#tensor-parallelism) (TP)

With very fast inter-node connectivity (e.g., NVLINK or NVSwitch) all three strategies (PP, ZeRO, TP) should result in 
similar performance. However, without these, PP will be faster than TP or ZeRO. The degree of TP may also 
make a difference. It's best to experiment with your specific setup to determine the most suitable strategy.

TP is almost always used within a single node. That is TP size <= GPUs per node.

**Case 3: Largest layer of your model does not fit onto a single GPU**

1. If you are not using ZeRO, you have to use TensorParallel (TP), because PipelineParallel (PP) alone won't be sufficient to accommodate the large layer.
2. If you are using ZeRO, additionally adopt techniques from the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one).

**Parallelization strategy for a multi-Node / multi-GPU setup**

* When you have fast inter-node connectivity (e.g., NVLINK or NVSwitch) consider using one of these options:

    1. ZeRO - as it requires close to no modifications to the model
    2. A combination of PipelineParallel(PP) with TensorParallel(TP) and DataParallel(DP) - this approach will result in fewer communications, but requires significant changes to the model

* When you have slow inter-node connectivity and still low on GPU memory:

    1. Employ a combination of DataParallel(DP) with PipelineParallel(PP), TensorParallel(TP), and ZeRO.

In the following sections of this guide we dig deeper into how these different parallelism methods work.

## Data Parallelism

Even with only 2 GPUs, you can readily leverage the accelerated training capabilities offered by PyTorch's built-in features, 
such as `DataParallel` (DP) and `DistributedDataParallel` (DDP). Note that 
[PyTorch documentation](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html) recommends to prefer 
`DistributedDataParallel` (DDP) over `DataParallel` (DP) for multi-GPU training as it works for all models.
Let's take a look at how these two methods work and what makes them different.


key: _node_content
string_value:
{"id_": "9daab811-bac6-4350-b397-a99814b4d21d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "47679084-01a3-4a08-9315-522e923de21a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "6b696a993028743669c1db7d105a69818b9a60ac643ccf3bb2bb192143b7a516", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a54f6548-2fca-47a4-9bcb-07ac4f8fc87e", "node_type": "1", "metadata": {}, "hash": "bf5757eb6a7e4fcfc7abc76d7017f285b7f99b1c77e398a6ae64f7d2a18eabe1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3174, "end_char_idx": 6470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
TP is almost always used within a single node. That is TP size <= GPUs per node.

**Case 3: Largest layer of your model does not fit onto a single GPU**

1. If you are not using ZeRO, you have to use TensorParallel (TP), because PipelineParallel (PP) alone won't be sufficient to accommodate the large layer.
2. If you are using ZeRO, additionally adopt techniques from the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one).

**Parallelization strategy for a multi-Node / multi-GPU setup**

* When you have fast inter-node connectivity (e.g., NVLINK or NVSwitch) consider using one of these options:

    1. ZeRO - as it requires close to no modifications to the model
    2. A combination of PipelineParallel(PP) with TensorParallel(TP) and DataParallel(DP) - this approach will result in fewer communications, but requires significant changes to the model

* When you have slow inter-node connectivity and still low on GPU memory:

    1. Employ a combination of DataParallel(DP) with PipelineParallel(PP), TensorParallel(TP), and ZeRO.

In the following sections of this guide we dig deeper into how these different parallelism methods work.

## Data Parallelism

Even with only 2 GPUs, you can readily leverage the accelerated training capabilities offered by PyTorch's built-in features, 
such as `DataParallel` (DP) and `DistributedDataParallel` (DDP). Note that 
[PyTorch documentation](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html) recommends to prefer 
`DistributedDataParallel` (DDP) over `DataParallel` (DP) for multi-GPU training as it works for all models.
Let's take a look at how these two methods work and what makes them different.

### DataParallel vs DistributedDataParallel

To understand the key differences in inter-GPU communication overhead between the two methods, let's review the processes per batch:

[DDP](https://pytorch.org/docs/master/notes/ddp.html):

- At the start time the main process replicates the model once from GPU 0 to the rest of GPUs
- Then for each batch:
   1. Each GPU directly consumes its mini-batch of data.
   2. During `backward`, once the local gradients are ready, they are averaged across all processes.

[DP](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html):

For each batch:
   1. GPU 0 reads the batch of data and then sends a mini-batch to each GPU.
   2. The up-to-date model is replicated from GPU 0 to each GPU. 
   3. `forward` is executed, and output from each GPU is sent to GPU 0 to compute the loss.
   4. The loss is distributed from GPU 0 to all GPUs, and `backward` is run. 
   5. Gradients from each GPU are sent to GPU 0 and averaged. 

Key differences include:
1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.
DDP copies data using [torch.distributed](https://pytorch.org/docs/master/distributed.html), while DP copies data within 
the process via Python threads (which introduces limitations associated with GIL). As a result, **`DistributedDataParallel` (DDP) is generally faster than `DataParallel` (DP)** unless you have slow GPU card inter-connectivity.
2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization.


key: _node_content
string_value:
{"id_": "a54f6548-2fca-47a4-9bcb-07ac4f8fc87e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9daab811-bac6-4350-b397-a99814b4d21d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "b1a6356698a0e08d26b9c013e52556c6de384f6863d8d15fc894d10c3bd9240d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "15afed5a-99eb-438e-9096-41e47b590233", "node_type": "1", "metadata": {}, "hash": "bebf73d163113ba00616cde8f96a8aaaa5c45c1401609b5cd9e2b64cd8d04f06", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4877, "end_char_idx": 7273, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### DataParallel vs DistributedDataParallel

To understand the key differences in inter-GPU communication overhead between the two methods, let's review the processes per batch:

[DDP](https://pytorch.org/docs/master/notes/ddp.html):

- At the start time the main process replicates the model once from GPU 0 to the rest of GPUs
- Then for each batch:
   1. Each GPU directly consumes its mini-batch of data.
   2. During `backward`, once the local gradients are ready, they are averaged across all processes.

[DP](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html):

For each batch:
   1. GPU 0 reads the batch of data and then sends a mini-batch to each GPU.
   2. The up-to-date model is replicated from GPU 0 to each GPU. 
   3. `forward` is executed, and output from each GPU is sent to GPU 0 to compute the loss.
   4. The loss is distributed from GPU 0 to all GPUs, and `backward` is run. 
   5. Gradients from each GPU are sent to GPU 0 and averaged. 

Key differences include:
1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.
DDP copies data using [torch.distributed](https://pytorch.org/docs/master/distributed.html), while DP copies data within 
the process via Python threads (which introduces limitations associated with GIL). As a result, **`DistributedDataParallel` (DDP) is generally faster than `DataParallel` (DP)** unless you have slow GPU card inter-connectivity.
2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization. 
3. DDP supports distributed training across multiple machines, whereas DP does not.

This is not an exhaustive list of differences between DP and DDP, however, other nuances are out of scope of this guide.
You can get a deeper understanding of these methods by reading this [article](https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/).

Let's illustrate the differences between DP and DDP with an experiment. We'll benchmark the differences between DP and 
DDP with an added context of NVLink presence:  

* Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (`NV2` in `nvidia-smi topo -m`).
* Software: `pytorch-1.8-to-be` + `cuda-11.0` / `transformers==4.3.0.dev0`.

To disable the NVLink feature on one of the benchmarks, we use `NCCL_P2P_DISABLE=1`.


key: _node_content
string_value:
{"id_": "15afed5a-99eb-438e-9096-41e47b590233", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a54f6548-2fca-47a4-9bcb-07ac4f8fc87e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "47aca6886d2272ced149346b81b34c2dc8d661a9ae3ff4a99f6be451a032eab9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e1f262c2-9179-4dc1-b248-9a96a21c982b", "node_type": "1", "metadata": {}, "hash": "85b9fa7888aa9d5271446a1e78e454e8f04022fe55412e29990e292f952b5cdb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5719, "end_char_idx": 8920, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4. The loss is distributed from GPU 0 to all GPUs, and `backward` is run. 
   5. Gradients from each GPU are sent to GPU 0 and averaged. 

Key differences include:
1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.
DDP copies data using [torch.distributed](https://pytorch.org/docs/master/distributed.html), while DP copies data within 
the process via Python threads (which introduces limitations associated with GIL). As a result, **`DistributedDataParallel` (DDP) is generally faster than `DataParallel` (DP)** unless you have slow GPU card inter-connectivity.
2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization. 
3. DDP supports distributed training across multiple machines, whereas DP does not.

This is not an exhaustive list of differences between DP and DDP, however, other nuances are out of scope of this guide.
You can get a deeper understanding of these methods by reading this [article](https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/).

Let's illustrate the differences between DP and DDP with an experiment. We'll benchmark the differences between DP and 
DDP with an added context of NVLink presence:  

* Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (`NV2` in `nvidia-smi topo -m`).
* Software: `pytorch-1.8-to-be` + `cuda-11.0` / `transformers==4.3.0.dev0`.

To disable the NVLink feature on one of the benchmarks, we use `NCCL_P2P_DISABLE=1`. 

Here is the benchmarking code and outputs:

**DP**

```bash
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
python examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 110.5948, 'train_samples_per_second': 1.808, 'epoch': 0.69}
```

**DDP w/ NVlink**

```bash
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
torchrun --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 101.9003, 'train_samples_per_second': 1.963, 'epoch': 0.69}
```

**DDP w/o NVlink**

```bash
rm -r /tmp/test-clm; NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1 \
torchrun --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 131.4367, 'train_samples_per_second': 1.522, 'epoch': 0.69}
```

Here are the same benchmarking results gathered in a table for convenience:

| Type   | NVlink | Time |
| :----- | -----  | ---: |
| 2:DP   | Y      | 110s |
| 2:DDP  | Y      | 101s |
| 2:DDP  | N      | 131s |

As you can see, in this case DP is ~10% slower than DDP with NVlink, but ~15% faster than DDP without NVlink.


key: _node_content
string_value:
{"id_": "e1f262c2-9179-4dc1-b248-9a96a21c982b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "15afed5a-99eb-438e-9096-41e47b590233", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "5670acada383953423275afbd73606c9ab38bdce9ce386abe45c3f5b64939b8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "48b86951-8e73-4363-b521-c44dcbbf3bb5", "node_type": "1", "metadata": {}, "hash": "378466ac7060365bbec651c1916cacb9d2cf1733d494ac3482c81a50a344fcaf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8921, "end_char_idx": 11691, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The real difference will depend on how much data each GPU needs to sync with the others - the more there is to sync, 
the more a slow link will impede the overall runtime.

## ZeRO Data Parallelism

ZeRO-powered data parallelism (ZeRO-DP) is illustrated in the following diagram from this [blog post](https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/).

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-zero.png" alt="DeepSpeed-Image-1"/>
 </div>

While it may appear complex, it is a very similar concept to `DataParallel` (DP). The difference is that instead of 
replicating the full model parameters, gradients and optimizer states, each GPU stores only a slice of it. Then, at 
run-time when the full layer parameters are needed just for the given layer, all GPUs synchronize to give each other 
parts that they miss.

To illustrate this idea, consider a simple model with 3 layers (La, Lb, and Lc), where each layer has 3 parameters. 
Layer La, for example, has weights a0, a1 and a2:

```
La | Lb | Lc
---|----|---
a0 | b0 | c0
a1 | b1 | c1
a2 | b2 | c2
```

If we have 3 GPUs, ZeRO-DP splits the model onto 3 GPUs like so:

```
GPU0:
La | Lb | Lc
---|----|---
a0 | b0 | c0

GPU1:
La | Lb | Lc
---|----|---
a1 | b1 | c1

GPU2:
La | Lb | Lc
---|----|---
a2 | b2 | c2
```

In a way, this is the same horizontal slicing as tensor parallelism, as opposed to Vertical 
slicing, where one puts whole layer-groups on different GPUs. Now let's see how this works: 

Each of these GPUs will get the usual mini-batch as it works in DP:

```
x0 => GPU0
x1 => GPU1
x2 => GPU2
```

The inputs are passed without modifications as if they would be processed by the original model.

First, the inputs get to the layer `La`. What happens at this point?

On GPU0: the x0 mini-batch requires the a0, a1, a2 parameters to do its forward path through the layer, but the GPU0 has only a0. 
It will get a1 from GPU1 and a2 from GPU2, bringing all the pieces of the model together.

In parallel, GPU1 gets another mini-batch - x1. GPU1 has the a1 parameter, but needs a0 and a2, so it gets those from GPU0 and GPU2.
Same happens to GPU2 that gets the mini-batch x2. It gets a0 and a1 from GPU0 and GPU1.

This way each of the 3 GPUs gets the full tensors reconstructed and makes a forward pass with its own mini-batch.
As soon as the calculation is done, the data that is no longer needed gets dropped - it's only used during the calculation. 
The reconstruction is done efficiently via a pre-fetch.

Then the whole process is repeated for layer Lb, then Lc forward-wise, and then backward Lc -> Lb -> La.


key: _node_content
string_value:
{"id_": "48b86951-8e73-4363-b521-c44dcbbf3bb5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e1f262c2-9179-4dc1-b248-9a96a21c982b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "ba2ec91a20f64213ab43a86eb2bef4b4b7405e40a77cd46c93edc379845389b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fd32670e-5e0b-49ec-99be-927e9c9e2eeb", "node_type": "1", "metadata": {}, "hash": "39be32304d025f8427f9a7fa59cceb63b5213d09d8f261d9ebdd90c787ccd727", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10536, "end_char_idx": 13444, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now let's see how this works: 

Each of these GPUs will get the usual mini-batch as it works in DP:

```
x0 => GPU0
x1 => GPU1
x2 => GPU2
```

The inputs are passed without modifications as if they would be processed by the original model.

First, the inputs get to the layer `La`. What happens at this point?

On GPU0: the x0 mini-batch requires the a0, a1, a2 parameters to do its forward path through the layer, but the GPU0 has only a0. 
It will get a1 from GPU1 and a2 from GPU2, bringing all the pieces of the model together.

In parallel, GPU1 gets another mini-batch - x1. GPU1 has the a1 parameter, but needs a0 and a2, so it gets those from GPU0 and GPU2.
Same happens to GPU2 that gets the mini-batch x2. It gets a0 and a1 from GPU0 and GPU1.

This way each of the 3 GPUs gets the full tensors reconstructed and makes a forward pass with its own mini-batch.
As soon as the calculation is done, the data that is no longer needed gets dropped - it's only used during the calculation. 
The reconstruction is done efficiently via a pre-fetch.

Then the whole process is repeated for layer Lb, then Lc forward-wise, and then backward Lc -> Lb -> La.

<Tip>

This mechanism is similar to an efficient group backpacking strategy: person A carries the tent, person B carries the stove,
and person C carries the axe. Each night they all share what they have with others and get from others what they don't have, 
and in the morning they pack up their allocated type of gear and continue on their way. This is what ZeRO DP/Sharded DDP is.
Compare this strategy to the simple one where each person has to carry their own tent, stove and axe (similar to 
DataParallel (DP and DDP) in PyTorch), which would be far more inefficient. 

</Tip>

While reading the literature on this topic you may encounter the following synonyms: Sharded, Partitioned.
If you pay close attention the way ZeRO partitions the model's weights - it looks very similar to tensor parallelism 
which will be discussed later. This is because it partitions/shards each layer's weights, unlike vertical model parallelism 
which is discussed next.

Implementations:

- [DeepSpeed](https://www.deepspeed.ai/tutorials/zero/) ZeRO-DP stages 1+2+3
- [`Accelerate` integration](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed) 
- [`transformers` integration](main_classes/trainer#trainer-integrations)

## From Naive Model Parallelism to Pipeline Parallelism

To explain Pipeline parallelism, we'll first look into Naive Model Parallelism (MP), also known as Vertical MP. This approach
involves distributing groups of model layers across multiple GPUs by assigning specific layers to specific GPUs with `.to()`. 
As data flows through these layers, it is moved to the same GPU as the layer, while the other layers remain untouched.

We refer to this Model parallelism as "Vertical" because of how models are typically visualized.


key: _node_content
string_value:
{"id_": "fd32670e-5e0b-49ec-99be-927e9c9e2eeb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "48b86951-8e73-4363-b521-c44dcbbf3bb5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "f94c80639eb959d39985d655b1eaddf316029daf6266529a439f0e02ad64eb28", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "880229f1-2f99-4c0a-9232-51d5713adfbc", "node_type": "1", "metadata": {}, "hash": "88288d70eac682fa5f9e040976555b7a9d20dc293bf1a83ffae53da30b85456a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11693, "end_char_idx": 14883, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

This mechanism is similar to an efficient group backpacking strategy: person A carries the tent, person B carries the stove,
and person C carries the axe. Each night they all share what they have with others and get from others what they don't have, 
and in the morning they pack up their allocated type of gear and continue on their way. This is what ZeRO DP/Sharded DDP is.
Compare this strategy to the simple one where each person has to carry their own tent, stove and axe (similar to 
DataParallel (DP and DDP) in PyTorch), which would be far more inefficient. 

</Tip>

While reading the literature on this topic you may encounter the following synonyms: Sharded, Partitioned.
If you pay close attention the way ZeRO partitions the model's weights - it looks very similar to tensor parallelism 
which will be discussed later. This is because it partitions/shards each layer's weights, unlike vertical model parallelism 
which is discussed next.

Implementations:

- [DeepSpeed](https://www.deepspeed.ai/tutorials/zero/) ZeRO-DP stages 1+2+3
- [`Accelerate` integration](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed) 
- [`transformers` integration](main_classes/trainer#trainer-integrations)

## From Naive Model Parallelism to Pipeline Parallelism

To explain Pipeline parallelism, we'll first look into Naive Model Parallelism (MP), also known as Vertical MP. This approach
involves distributing groups of model layers across multiple GPUs by assigning specific layers to specific GPUs with `.to()`. 
As data flows through these layers, it is moved to the same GPU as the layer, while the other layers remain untouched.

We refer to this Model parallelism as "Vertical" because of how models are typically visualized. For example, the 
following diagram shows an 8-layer model split vertically into two slices, placing layers 0-3 onto 
GPU0 and 4-7 to GPU1:

```
================
| Layer |      |
|   0   |      |
|   1   | GPU0 |
|   2   |      |
|   3   |      |
================
| Layer |      |
|   4   |      |
|   5   | GPU1 |
|   6   |      |
|   7   |      |
================
```

In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data 
from layer 3 to 4 requires moving it from GPU0 to GPU1, introducing a communication overhead. If the participating 
GPUs are on the same compute node (e.g. same physical machine) this copying is fast, but if the GPUs are distributed 
across different compute nodes (e.g. multiple machines), the communication overhead could be substantially greater.

Following that, layers 4 to 7 work as they would in the original model. Upon completion of the 7th layer, there is often 
a need to send the data back to layer 0 where the labels are (or alternatively send the labels to the last layer). Now the loss can be 
computed and the optimizer can do its work.

Naive Model Parallelism comes several shortcomings:
- **All but one GPU are idle at any given moment**: if 4 GPUs are used, it's nearly identical to quadrupling the amount of memory of a single GPU, and ignoring the rest of the hardware. 
- **Overhead in data transfer between devices**:  E.g.


key: _node_content
string_value:
{"id_": "880229f1-2f99-4c0a-9232-51d5713adfbc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fd32670e-5e0b-49ec-99be-927e9c9e2eeb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "23084d12f13ed7910f74aca49a5628c97634c087c9fbab4687d037a02bd4e8cf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fc51d7c7-3c9b-4725-af4a-4a1ff26e033b", "node_type": "1", "metadata": {}, "hash": "f21456a3a798fa9cd866e2e8e3633b38883697b0ac667f86952cf9ab85ef6a51", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13348, "end_char_idx": 16360, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We refer to this Model parallelism as "Vertical" because of how models are typically visualized. For example, the 
following diagram shows an 8-layer model split vertically into two slices, placing layers 0-3 onto 
GPU0 and 4-7 to GPU1:

```
================
| Layer |      |
|   0   |      |
|   1   | GPU0 |
|   2   |      |
|   3   |      |
================
| Layer |      |
|   4   |      |
|   5   | GPU1 |
|   6   |      |
|   7   |      |
================
```

In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data 
from layer 3 to 4 requires moving it from GPU0 to GPU1, introducing a communication overhead. If the participating 
GPUs are on the same compute node (e.g. same physical machine) this copying is fast, but if the GPUs are distributed 
across different compute nodes (e.g. multiple machines), the communication overhead could be substantially greater.

Following that, layers 4 to 7 work as they would in the original model. Upon completion of the 7th layer, there is often 
a need to send the data back to layer 0 where the labels are (or alternatively send the labels to the last layer). Now the loss can be 
computed and the optimizer can do its work.

Naive Model Parallelism comes several shortcomings:
- **All but one GPU are idle at any given moment**: if 4 GPUs are used, it's nearly identical to quadrupling the amount of memory of a single GPU, and ignoring the rest of the hardware. 
- **Overhead in data transfer between devices**:  E.g. 4x 6GB cards will be able to accommodate the same size as 1x 24GB card using naive MP, but a single 24GB card will complete the training faster, because it doesn't have the data copying overhead. But, say, if you have 40GB cards and need to fit a 45GB model you can with 4x 40GB cards (but barely because of the gradient and optimizer states)
- **Copying shared embeddings**: Shared embeddings may need to get copied back and forth between GPUs.

Now that you are familiar with how the naive approach to model parallelism works and its shortcomings, let's look at Pipeline Parallelism (PP).
PP is almost identical to a naive MP, but it solves the GPU idling problem by chunking the incoming batch into micro-batches 
and artificially creating a pipeline, which allows different GPUs to concurrently participate in the computation process.

The following illustration from the [GPipe paper](https://ai.googleblog.com/2019/03/introducing-gpipe-open-source-library.html) 
shows the naive MP on the top, and PP on the bottom:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-gpipe-bubble.png" alt="MP vs PP"/>
</div>

At the bottom of the diagram, you can observe that the Pipeline Parallelism (PP) approach minimizes the number of idle 
GPU zones, referred to as 'bubbles'. Both parts of the diagram show a parallelism level of degree 4, meaning that 4 GPUs 
are involved in the pipeline.


key: _node_content
string_value:
{"id_": "fc51d7c7-3c9b-4725-af4a-4a1ff26e033b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "880229f1-2f99-4c0a-9232-51d5713adfbc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "7620cd2a4f2305f994b83300dc7d579bcd44dfb60c72420bbb578e7e55c6336a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "acce0d79-da98-4b73-a800-32d153b615f5", "node_type": "1", "metadata": {}, "hash": "6f6b40adaaee90b28b6ff7e08d62b460cbe23ae996936bef9639f66f0631b33b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14829, "end_char_idx": 17787, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- **Overhead in data transfer between devices**:  E.g. 4x 6GB cards will be able to accommodate the same size as 1x 24GB card using naive MP, but a single 24GB card will complete the training faster, because it doesn't have the data copying overhead. But, say, if you have 40GB cards and need to fit a 45GB model you can with 4x 40GB cards (but barely because of the gradient and optimizer states)
- **Copying shared embeddings**: Shared embeddings may need to get copied back and forth between GPUs.

Now that you are familiar with how the naive approach to model parallelism works and its shortcomings, let's look at Pipeline Parallelism (PP).
PP is almost identical to a naive MP, but it solves the GPU idling problem by chunking the incoming batch into micro-batches 
and artificially creating a pipeline, which allows different GPUs to concurrently participate in the computation process.

The following illustration from the [GPipe paper](https://ai.googleblog.com/2019/03/introducing-gpipe-open-source-library.html) 
shows the naive MP on the top, and PP on the bottom:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-gpipe-bubble.png" alt="MP vs PP"/>
</div>

At the bottom of the diagram, you can observe that the Pipeline Parallelism (PP) approach minimizes the number of idle 
GPU zones, referred to as 'bubbles'. Both parts of the diagram show a parallelism level of degree 4, meaning that 4 GPUs 
are involved in the pipeline. You can see that there's a forward path of 4 pipe stages (F0, F1, F2 and F3) followed by 
a backward path in reverse order (B3, B2, B1, and B0).

PP introduces a new hyperparameter to tune - `chunks`, which determines how many data chunks are sent in a sequence 
through the same pipe stage. For example, in the bottom diagram you can see `chunks=4`. GPU0 performs the same 
forward path on chunk 0, 1, 2 and 3 (F0,0, F0,1, F0,2, F0,3) and then it waits for other GPUs to do complete their work. 
Only when the other GPUs begin to complete their work, GPU0 starts to work again doing the backward path for chunks 
3, 2, 1 and 0 (B0,3, B0,2, B0,1, B0,0).

Note that this is the same concept as gradient accumulation steps. PyTorch uses `chunks`, while DeepSpeed refers 
to the same hyperparameter as gradient accumulation steps.

Because of the chunks, PP introduces the notion of micro-batches (MBS). DP splits the global data batch size into 
mini-batches, so if you have a DP degree of 4, a global batch size of 1024 gets split up into 4 mini-batches of 
256 each (1024/4). And if the number of `chunks` (or GAS) is 32 we end up with a micro-batch size of 8 (256/32). Each 
Pipeline stage works with a single micro-batch at a time. To calculate the global batch size of the DP + PP setup, 
use the formula: `mbs * chunks * dp_degree` (`8 * 32 * 4 = 1024`).
With `chunks=1` you end up with the naive MP, which is inefficient.


key: _node_content
string_value:
{"id_": "acce0d79-da98-4b73-a800-32d153b615f5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fc51d7c7-3c9b-4725-af4a-4a1ff26e033b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "3e906ec450eeb649c122789b632a3e5695c37c4b96cac42b54dd3c4635e0a53f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a14dab91-0eb3-4909-916f-581878333083", "node_type": "1", "metadata": {}, "hash": "4bae61f592bceaec4da24c3c774a5e8315cba854e4f1fb2708c619d07aef88d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16507, "end_char_idx": 19513, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
PP introduces a new hyperparameter to tune - `chunks`, which determines how many data chunks are sent in a sequence 
through the same pipe stage. For example, in the bottom diagram you can see `chunks=4`. GPU0 performs the same 
forward path on chunk 0, 1, 2 and 3 (F0,0, F0,1, F0,2, F0,3) and then it waits for other GPUs to do complete their work. 
Only when the other GPUs begin to complete their work, GPU0 starts to work again doing the backward path for chunks 
3, 2, 1 and 0 (B0,3, B0,2, B0,1, B0,0).

Note that this is the same concept as gradient accumulation steps. PyTorch uses `chunks`, while DeepSpeed refers 
to the same hyperparameter as gradient accumulation steps.

Because of the chunks, PP introduces the notion of micro-batches (MBS). DP splits the global data batch size into 
mini-batches, so if you have a DP degree of 4, a global batch size of 1024 gets split up into 4 mini-batches of 
256 each (1024/4). And if the number of `chunks` (or GAS) is 32 we end up with a micro-batch size of 8 (256/32). Each 
Pipeline stage works with a single micro-batch at a time. To calculate the global batch size of the DP + PP setup, 
use the formula: `mbs * chunks * dp_degree` (`8 * 32 * 4 = 1024`).
With `chunks=1` you end up with the naive MP, which is inefficient. With a large `chunks` value you end up with 
tiny micro-batch sizes which is also inefficient. For this reason, we encourage to experiment with the `chunks` value to 
find the one that leads to the most efficient GPUs utilization.

You may notice a bubble of "dead" time on the diagram that can't be parallelized because the last `forward` stage 
has to wait for `backward` to complete the pipeline. The purpose of finding the best value for `chunks` is to enable a high 
concurrent GPU utilization across all participating GPUs which translates to minimizing the size of the bubble.

Pipeline API solutions have been implemented in:
- PyTorch
- DeepSpeed
- Megatron-LM

These come with some shortcomings:
- They have to modify the model quite heavily, because Pipeline requires one to rewrite the normal flow of modules into a `nn.Sequential` sequence of the same, which may require changes to the design of the model.
- Currently the Pipeline API is very restricted. If you had a bunch of Python variables being passed in the very first stage of the Pipeline, you will have to find a way around it. Currently, the pipeline interface requires either a single Tensor or a tuple of Tensors as the only input and output. These tensors must have a batch size as the very first dimension, since pipeline is going to chunk the mini batch into micro-batches. Possible improvements are being discussed here https://github.com/pytorch/pytorch/pull/50693
- Conditional control flow at the level of pipe stages is not possible - e.g., Encoder-Decoder models like T5 require special workarounds to handle a conditional encoder stage.
- They have to arrange each layer so that the output of one layer becomes an input to the other layer.


key: _node_content
string_value:
{"id_": "a14dab91-0eb3-4909-916f-581878333083", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "acce0d79-da98-4b73-a800-32d153b615f5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "decb13d3622f891d4f86e59839e97be00432f168b82fdbbcd5e56ee7e98b7cdf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5b9ae882-a5df-4188-9dbd-962d17a137f4", "node_type": "1", "metadata": {}, "hash": "c195026b5a467d8dff6cde01624b0c46f0b525bc16611e79a1a01f53ce005fc3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17720, "end_char_idx": 20849, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With `chunks=1` you end up with the naive MP, which is inefficient. With a large `chunks` value you end up with 
tiny micro-batch sizes which is also inefficient. For this reason, we encourage to experiment with the `chunks` value to 
find the one that leads to the most efficient GPUs utilization.

You may notice a bubble of "dead" time on the diagram that can't be parallelized because the last `forward` stage 
has to wait for `backward` to complete the pipeline. The purpose of finding the best value for `chunks` is to enable a high 
concurrent GPU utilization across all participating GPUs which translates to minimizing the size of the bubble.

Pipeline API solutions have been implemented in:
- PyTorch
- DeepSpeed
- Megatron-LM

These come with some shortcomings:
- They have to modify the model quite heavily, because Pipeline requires one to rewrite the normal flow of modules into a `nn.Sequential` sequence of the same, which may require changes to the design of the model.
- Currently the Pipeline API is very restricted. If you had a bunch of Python variables being passed in the very first stage of the Pipeline, you will have to find a way around it. Currently, the pipeline interface requires either a single Tensor or a tuple of Tensors as the only input and output. These tensors must have a batch size as the very first dimension, since pipeline is going to chunk the mini batch into micro-batches. Possible improvements are being discussed here https://github.com/pytorch/pytorch/pull/50693
- Conditional control flow at the level of pipe stages is not possible - e.g., Encoder-Decoder models like T5 require special workarounds to handle a conditional encoder stage.
- They have to arrange each layer so that the output of one layer becomes an input to the other layer.

More recent solutions include:
- Varuna
- Sagemaker

We have not experimented with Varuna and SageMaker but their papers report that they have overcome the list of problems 
mentioned above and that they require smaller changes to the user's model.

Implementations:
- [PyTorch](https://pytorch.org/docs/stable/pipeline.html) (initial support in pytorch-1.8, and progressively getting improved in 1.9 and more so in 1.10). Some [examples](https://github.com/pytorch/pytorch/blob/master/benchmarks/distributed/pipeline/pipe.py)
- [DeepSpeed](https://www.deepspeed.ai/tutorials/pipeline/)
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation - no API.
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.
- [OSLO](https://github.com/tunib-ai/oslo) - this is implemented based on the Hugging Face Transformers.

🤗 Transformers status: as of this writing none of the models supports full-PP. GPT2 and T5 models have naive MP support. 
The main obstacle is being unable to convert the models to `nn.Sequential` and have all the inputs to be Tensors. This 
is because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.


key: _node_content
string_value:
{"id_": "5b9ae882-a5df-4188-9dbd-962d17a137f4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a14dab91-0eb3-4909-916f-581878333083", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "a19f44e0e1a8d24879d641314db2a60742cff32f25c14c69eefbbfbdfd7cfe3e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "11229f7f-a48c-431e-8eb2-80d38d6782aa", "node_type": "1", "metadata": {}, "hash": "7ba53e8b46a5a74f990f09411f006cad4726e6002acb19f0a2d65cf95280ab50", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19411, "end_char_idx": 22419, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- They have to arrange each layer so that the output of one layer becomes an input to the other layer.

More recent solutions include:
- Varuna
- Sagemaker

We have not experimented with Varuna and SageMaker but their papers report that they have overcome the list of problems 
mentioned above and that they require smaller changes to the user's model.

Implementations:
- [PyTorch](https://pytorch.org/docs/stable/pipeline.html) (initial support in pytorch-1.8, and progressively getting improved in 1.9 and more so in 1.10). Some [examples](https://github.com/pytorch/pytorch/blob/master/benchmarks/distributed/pipeline/pipe.py)
- [DeepSpeed](https://www.deepspeed.ai/tutorials/pipeline/)
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation - no API.
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.
- [OSLO](https://github.com/tunib-ai/oslo) - this is implemented based on the Hugging Face Transformers.

🤗 Transformers status: as of this writing none of the models supports full-PP. GPT2 and T5 models have naive MP support. 
The main obstacle is being unable to convert the models to `nn.Sequential` and have all the inputs to be Tensors. This 
is because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.

DeepSpeed and Megatron-LM integrations are available in [🤗 Accelerate](https://huggingface.co/docs/accelerate/main/en/usage_guides/deepspeed)

Other approaches:

DeepSpeed, Varuna and SageMaker use the concept of an [Interleaved Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-core-features.html)

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-sagemaker-interleaved-pipeline.png" alt="Interleaved pipeline execution"/>
</div>

Here the bubble (idle time) is further minimized by prioritizing backward passes. Varuna further attempts to improve the 
schedule by using simulations to discover the most efficient scheduling.

OSLO has pipeline parallelism implementation based on the Transformers without `nn.Sequential` conversion.

## Tensor Parallelism

In Tensor Parallelism, each GPU processes a slice of a tensor and only aggregates the full tensor for operations requiring it.
To describe this method, this section of the guide relies on the concepts and diagrams from the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) 
paper: [Efficient Large-Scale Language Model Training on GPU Clusters](https://arxiv.org/abs/2104.04473).

The main building block of any transformer is a fully connected `nn.Linear` followed by a nonlinear activation `GeLU`.
The dot dot-product part of it, following the Megatron's paper notation, can be written as `Y = GeLU(XA)`, where `X` is 
an input vector, `Y` is the output vector, and `A` is the weight matrix.


key: _node_content
string_value:
{"id_": "11229f7f-a48c-431e-8eb2-80d38d6782aa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5b9ae882-a5df-4188-9dbd-962d17a137f4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "e069ddd5bab6d1640b7097a215d5dff5376b313b6458389bb95ebd931c53ba3e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ba26e22-135e-47d8-9740-e11258c95e06", "node_type": "1", "metadata": {}, "hash": "f34bf66b9ce0fd630ec6e95cc081ef3e3b5570135166177ef323ed45223e8d17", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20699, "end_char_idx": 23832, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This 
is because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.

DeepSpeed and Megatron-LM integrations are available in [🤗 Accelerate](https://huggingface.co/docs/accelerate/main/en/usage_guides/deepspeed)

Other approaches:

DeepSpeed, Varuna and SageMaker use the concept of an [Interleaved Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-core-features.html)

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-sagemaker-interleaved-pipeline.png" alt="Interleaved pipeline execution"/>
</div>

Here the bubble (idle time) is further minimized by prioritizing backward passes. Varuna further attempts to improve the 
schedule by using simulations to discover the most efficient scheduling.

OSLO has pipeline parallelism implementation based on the Transformers without `nn.Sequential` conversion.

## Tensor Parallelism

In Tensor Parallelism, each GPU processes a slice of a tensor and only aggregates the full tensor for operations requiring it.
To describe this method, this section of the guide relies on the concepts and diagrams from the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) 
paper: [Efficient Large-Scale Language Model Training on GPU Clusters](https://arxiv.org/abs/2104.04473).

The main building block of any transformer is a fully connected `nn.Linear` followed by a nonlinear activation `GeLU`.
The dot dot-product part of it, following the Megatron's paper notation, can be written as `Y = GeLU(XA)`, where `X` is 
an input vector, `Y` is the output vector, and `A` is the weight matrix.

If we look at the computation in matrix form, you can see how the matrix multiplication can be split between multiple GPUs:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_gemm.png" alt="Parallel GEMM"/>
</div>

If we split the weight matrix `A` column-wise across `N` GPUs and perform matrix multiplications `XA_1` through `XA_n` in parallel, 
then we will end up with `N` output vectors `Y_1, Y_2, ..., Y_n` which can be fed into `GeLU` independently:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-independent-gelu.png" alt="Independent GeLU"/>
</div>

Using this principle, we can update a multi-layer perceptron of arbitrary depth, without the need for any synchronization 
between GPUs until the very end, where we need to reconstruct the output vector from shards. The Megatron-LM paper authors 
provide a helpful illustration for that:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_shard_processing.png" alt="Parallel shard processing"/>
</div>

Parallelizing the multi-headed attention layers is even simpler, since they are already inherently parallel, due to having 
multiple independent heads!


key: _node_content
string_value:
{"id_": "1ba26e22-135e-47d8-9740-e11258c95e06", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "11229f7f-a48c-431e-8eb2-80d38d6782aa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "d97a9fe91b2e878decbce2bcfc0d7116f08be5854ec9f19898139f3fd62e049a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5b5db4f1-0d7a-42e9-acc6-8c84fedd6655", "node_type": "1", "metadata": {}, "hash": "1579173a19c4dc268652b0e186f5fbd1bed89637831373171cd6839dfb99a8e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22226, "end_char_idx": 25408, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The dot dot-product part of it, following the Megatron's paper notation, can be written as `Y = GeLU(XA)`, where `X` is 
an input vector, `Y` is the output vector, and `A` is the weight matrix.

If we look at the computation in matrix form, you can see how the matrix multiplication can be split between multiple GPUs:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_gemm.png" alt="Parallel GEMM"/>
</div>

If we split the weight matrix `A` column-wise across `N` GPUs and perform matrix multiplications `XA_1` through `XA_n` in parallel, 
then we will end up with `N` output vectors `Y_1, Y_2, ..., Y_n` which can be fed into `GeLU` independently:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-independent-gelu.png" alt="Independent GeLU"/>
</div>

Using this principle, we can update a multi-layer perceptron of arbitrary depth, without the need for any synchronization 
between GPUs until the very end, where we need to reconstruct the output vector from shards. The Megatron-LM paper authors 
provide a helpful illustration for that:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_shard_processing.png" alt="Parallel shard processing"/>
</div>

Parallelizing the multi-headed attention layers is even simpler, since they are already inherently parallel, due to having 
multiple independent heads!

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_self_attention.png" alt="Parallel self-attention"/>
</div>

Special considerations: TP requires very fast network, and therefore it's not advisable to do TP across more than one node. 
Practically, if a node has 4 GPUs, the highest TP degree is therefore 4. If you need a TP degree of 8, you need to use
nodes that have at least 8 GPUs.

This section is based on the original much more [detailed TP overview](https://github.com/huggingface/transformers/issues/10321#issuecomment-783543530).
by [@anton-l](https://github.com/anton-l).

Alternative names:
- DeepSpeed calls it [tensor slicing](https://www.deepspeed.ai/training/#model-parallelism)

Implementations:
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation, as it's very model-specific
- [parallelformers](https://github.com/tunib-ai/parallelformers) (only inference at the moment)
- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.
- [OSLO](https://github.com/tunib-ai/oslo) has the tensor parallelism implementation based on the Transformers.

SageMaker combines TP with DP for a more efficient processing.

🤗 Transformers status:
- core: not yet implemented in the core
- but if you want inference [parallelformers](https://github.com/tunib-ai/parallelformers) provides this support for most of our models. So until this is implemented in the core you can use theirs.


key: _node_content
string_value:
{"id_": "5b5db4f1-0d7a-42e9-acc6-8c84fedd6655", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1ba26e22-135e-47d8-9740-e11258c95e06", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "8558903d67dcc07964f69de962ba029ef82aa05e163ed36a26844674b4a1c49a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f783ed12-5f26-45f6-907b-41cd6097b56e", "node_type": "1", "metadata": {}, "hash": "9daaac50d1e5dab161b4257cd75157a8a59554015a1c7a96a3cc7e93ad80c212", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23834, "end_char_idx": 26536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_self_attention.png" alt="Parallel self-attention"/>
</div>

Special considerations: TP requires very fast network, and therefore it's not advisable to do TP across more than one node. 
Practically, if a node has 4 GPUs, the highest TP degree is therefore 4. If you need a TP degree of 8, you need to use
nodes that have at least 8 GPUs.

This section is based on the original much more [detailed TP overview](https://github.com/huggingface/transformers/issues/10321#issuecomment-783543530).
by [@anton-l](https://github.com/anton-l).

Alternative names:
- DeepSpeed calls it [tensor slicing](https://www.deepspeed.ai/training/#model-parallelism)

Implementations:
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation, as it's very model-specific
- [parallelformers](https://github.com/tunib-ai/parallelformers) (only inference at the moment)
- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.
- [OSLO](https://github.com/tunib-ai/oslo) has the tensor parallelism implementation based on the Transformers.

SageMaker combines TP with DP for a more efficient processing.

🤗 Transformers status:
- core: not yet implemented in the core
- but if you want inference [parallelformers](https://github.com/tunib-ai/parallelformers) provides this support for most of our models. So until this is implemented in the core you can use theirs. And hopefully training mode will be supported too.
- Deepspeed-Inference also supports our BERT, GPT-2, and GPT-Neo models in their super-fast CUDA-kernel-based inference mode, see more [here](https://www.deepspeed.ai/tutorials/inference-tutorial/)

🤗 Accelerate integrates with [TP from Megatron-LM](https://huggingface.co/docs/accelerate/v0.23.0/en/usage_guides/megatron_lm).

## Data Parallelism + Pipeline Parallelism

The following diagram from the DeepSpeed [pipeline tutorial](https://www.deepspeed.ai/tutorials/pipeline/) demonstrates 
how one can combine DP with PP.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-zero-dp-pp.png" alt="DP + PP-2d"/>
</div>

Here it's important to see how DP rank 0 doesn't see GPU2 and DP rank 1 doesn't see GPU3. To DP there is just GPUs 0 
and 1 where it feeds data as if there were just 2 GPUs. GPU0 "secretly" offloads some of its load to GPU2 using PP. 
And GPU1 does the same by enlisting GPU3 to its aid.

Since each dimension requires at least 2 GPUs, here you'd need at least 4 GPUs.


key: _node_content
string_value:
{"id_": "f783ed12-5f26-45f6-907b-41cd6097b56e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5b5db4f1-0d7a-42e9-acc6-8c84fedd6655", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "2402fb110c9e148b965ec754f2bbfbf9395c12d647fa6a39012b04b9e204fa3e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "271b7fc1-af02-43b0-9b13-1aaeef059994", "node_type": "1", "metadata": {}, "hash": "df1b6aab547fc51aab921e9791cf770aab086fc4c079af09e1964ca4389510c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25084, "end_char_idx": 27959, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
SageMaker combines TP with DP for a more efficient processing.

🤗 Transformers status:
- core: not yet implemented in the core
- but if you want inference [parallelformers](https://github.com/tunib-ai/parallelformers) provides this support for most of our models. So until this is implemented in the core you can use theirs. And hopefully training mode will be supported too.
- Deepspeed-Inference also supports our BERT, GPT-2, and GPT-Neo models in their super-fast CUDA-kernel-based inference mode, see more [here](https://www.deepspeed.ai/tutorials/inference-tutorial/)

🤗 Accelerate integrates with [TP from Megatron-LM](https://huggingface.co/docs/accelerate/v0.23.0/en/usage_guides/megatron_lm).

## Data Parallelism + Pipeline Parallelism

The following diagram from the DeepSpeed [pipeline tutorial](https://www.deepspeed.ai/tutorials/pipeline/) demonstrates 
how one can combine DP with PP.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-zero-dp-pp.png" alt="DP + PP-2d"/>
</div>

Here it's important to see how DP rank 0 doesn't see GPU2 and DP rank 1 doesn't see GPU3. To DP there is just GPUs 0 
and 1 where it feeds data as if there were just 2 GPUs. GPU0 "secretly" offloads some of its load to GPU2 using PP. 
And GPU1 does the same by enlisting GPU3 to its aid.

Since each dimension requires at least 2 GPUs, here you'd need at least 4 GPUs.

Implementations:
- [DeepSpeed](https://github.com/microsoft/DeepSpeed)
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972)
- [OSLO](https://github.com/tunib-ai/oslo)

🤗 Transformers status: not yet implemented

## Data Parallelism + Pipeline Parallelism + Tensor Parallelism

To get an even more efficient training a 3D parallelism is used where PP is combined with TP and DP. This can be seen in the following diagram.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-deepspeed-3d.png" alt="dp-pp-tp-3d"/>
</div>

This diagram is from a blog post [3D parallelism: Scaling to trillion-parameter models](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/), which is a good read as well.

Since each dimension requires at least 2 GPUs, here you'd need at least 8 GPUs.

Implementations:
- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - DeepSpeed also includes an even more efficient DP, which they call ZeRO-DP.
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972)
- [OSLO](https://github.com/tunib-ai/oslo)

🤗 Transformers status: not yet implemented, since we have no PP and TP.


key: _node_content
string_value:
{"id_": "271b7fc1-af02-43b0-9b13-1aaeef059994", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f783ed12-5f26-45f6-907b-41cd6097b56e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "b5c7201c4d33f54ecb0953baf9e3efdd912a5a19406cec33fa72a6b55e059ff1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5670af0d-319c-4e4c-a635-f2d179b75a9a", "node_type": "1", "metadata": {}, "hash": "798e89309d62d4c9ddb2bb15f18157f28d13b2ac30c4b68483856224c3d5be9e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26538, "end_char_idx": 29515, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Implementations:
- [DeepSpeed](https://github.com/microsoft/DeepSpeed)
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972)
- [OSLO](https://github.com/tunib-ai/oslo)

🤗 Transformers status: not yet implemented

## Data Parallelism + Pipeline Parallelism + Tensor Parallelism

To get an even more efficient training a 3D parallelism is used where PP is combined with TP and DP. This can be seen in the following diagram.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-deepspeed-3d.png" alt="dp-pp-tp-3d"/>
</div>

This diagram is from a blog post [3D parallelism: Scaling to trillion-parameter models](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/), which is a good read as well.

Since each dimension requires at least 2 GPUs, here you'd need at least 8 GPUs.

Implementations:
- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - DeepSpeed also includes an even more efficient DP, which they call ZeRO-DP.
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972)
- [OSLO](https://github.com/tunib-ai/oslo)

🤗 Transformers status: not yet implemented, since we have no PP and TP.

## ZeRO Data Parallelism + Pipeline Parallelism + Tensor Parallelism

One of the main features of DeepSpeed is ZeRO, which is a super-scalable extension of DP. It has already been 
discussed in [ZeRO Data Parallelism](#zero-data-parallelism). Normally it's a standalone feature that doesn't require PP or TP. 
But it can be combined with PP and TP.

When ZeRO-DP is combined with PP (and optionally TP) it typically enables only ZeRO stage 1 (optimizer sharding).

While it's theoretically possible to use ZeRO stage 2 (gradient sharding) with Pipeline Parallelism, it will have negative 
performance impacts. There would need to be an additional reduce-scatter collective for every micro-batch to aggregate 
the gradients before sharding, which adds a potentially significant communication overhead. By nature of Pipeline Parallelism, 
small micro-batches are used and instead the focus is on trying to balance arithmetic intensity (micro-batch size) with
minimizing the Pipeline bubble (number of micro-batches). Therefore those communication costs are going to impact the performance.

In addition, there are already fewer layers than normal due to PP and so the memory savings won't be huge. PP already 
reduces gradient size by ``1/PP``, and so gradient sharding savings on top of that are less significant than pure DP.

ZeRO stage 3 is not a good choice either for the same reason - more inter-node communications required.

And since we have ZeRO, the other benefit is ZeRO-Offload. Since this is stage 1 optimizer states can be offloaded to CPU.


key: _node_content
string_value:
{"id_": "5670af0d-319c-4e4c-a635-f2d179b75a9a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "271b7fc1-af02-43b0-9b13-1aaeef059994", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "7c5167ef9a5da1263d4449bcbb2850224245c015cb3dc0b7be9057afd754b2cf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "65dd47e0-ebd5-497b-92ec-6c46f2c48ae2", "node_type": "1", "metadata": {}, "hash": "44d101216853fa6795481b8a4b9604ed5895bc73e6c5de3023a821484adedad3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27961, "end_char_idx": 31196, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## ZeRO Data Parallelism + Pipeline Parallelism + Tensor Parallelism

One of the main features of DeepSpeed is ZeRO, which is a super-scalable extension of DP. It has already been 
discussed in [ZeRO Data Parallelism](#zero-data-parallelism). Normally it's a standalone feature that doesn't require PP or TP. 
But it can be combined with PP and TP.

When ZeRO-DP is combined with PP (and optionally TP) it typically enables only ZeRO stage 1 (optimizer sharding).

While it's theoretically possible to use ZeRO stage 2 (gradient sharding) with Pipeline Parallelism, it will have negative 
performance impacts. There would need to be an additional reduce-scatter collective for every micro-batch to aggregate 
the gradients before sharding, which adds a potentially significant communication overhead. By nature of Pipeline Parallelism, 
small micro-batches are used and instead the focus is on trying to balance arithmetic intensity (micro-batch size) with
minimizing the Pipeline bubble (number of micro-batches). Therefore those communication costs are going to impact the performance.

In addition, there are already fewer layers than normal due to PP and so the memory savings won't be huge. PP already 
reduces gradient size by ``1/PP``, and so gradient sharding savings on top of that are less significant than pure DP.

ZeRO stage 3 is not a good choice either for the same reason - more inter-node communications required.

And since we have ZeRO, the other benefit is ZeRO-Offload. Since this is stage 1 optimizer states can be offloaded to CPU.

Implementations:
- [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed) and [Megatron-Deepspeed from BigScience](https://github.com/bigscience-workshop/Megatron-DeepSpeed), which is the fork of the former repo.
- [OSLO](https://github.com/tunib-ai/oslo)

Important papers:

- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](
https://arxiv.org/abs/2201.11990)

🤗 Transformers status: not yet implemented, since we have no PP and TP.

## FlexFlow

[FlexFlow](https://github.com/flexflow/FlexFlow) also solves the parallelization problem in a slightly different approach.

Paper: ["Beyond Data and Model Parallelism for Deep Neural Networks" by Zhihao Jia, Matei Zaharia, Alex Aiken](https://arxiv.org/abs/1807.05358)

It performs a sort of 4D Parallelism over Sample-Operator-Attribute-Parameter.

1. Sample = Data Parallelism (sample-wise parallel)
2. Operator = Parallelize a single operation into several sub-operations
3. Attribute = Data Parallelism (length-wise parallel)
4. Parameter = Model Parallelism (regardless of dimension - horizontal or vertical)

Examples:
* Sample

Let's take 10 batches of sequence length 512. If we parallelize them by sample dimension into 2 devices, we get 10 x 512 which becomes be 5 x 2 x 512.

* Operator

If we perform layer normalization, we compute std first and mean second, and then we can normalize data. 
Operator parallelism allows computing std and mean in parallel. So if we parallelize them by operator dimension into 2 
devices (cuda:0, cuda:1), first we copy input data into both devices, and cuda:0 computes std, cuda:1 computes mean at the same time.


key: _node_content
string_value:
{"id_": "65dd47e0-ebd5-497b-92ec-6c46f2c48ae2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5670af0d-319c-4e4c-a635-f2d179b75a9a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "ca19081580df4c369c168f64b64631d690664a08eba965819c02ca0febbef29a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b8a40a0-d996-4cfe-ba92-0287b2bd216d", "node_type": "1", "metadata": {}, "hash": "464d90f681f30c534c12c07ead6a8428b748112b55decfe75eb4c0b554638191", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29745, "end_char_idx": 32980, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [OSLO](https://github.com/tunib-ai/oslo)

Important papers:

- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](
https://arxiv.org/abs/2201.11990)

🤗 Transformers status: not yet implemented, since we have no PP and TP.

## FlexFlow

[FlexFlow](https://github.com/flexflow/FlexFlow) also solves the parallelization problem in a slightly different approach.

Paper: ["Beyond Data and Model Parallelism for Deep Neural Networks" by Zhihao Jia, Matei Zaharia, Alex Aiken](https://arxiv.org/abs/1807.05358)

It performs a sort of 4D Parallelism over Sample-Operator-Attribute-Parameter.

1. Sample = Data Parallelism (sample-wise parallel)
2. Operator = Parallelize a single operation into several sub-operations
3. Attribute = Data Parallelism (length-wise parallel)
4. Parameter = Model Parallelism (regardless of dimension - horizontal or vertical)

Examples:
* Sample

Let's take 10 batches of sequence length 512. If we parallelize them by sample dimension into 2 devices, we get 10 x 512 which becomes be 5 x 2 x 512.

* Operator

If we perform layer normalization, we compute std first and mean second, and then we can normalize data. 
Operator parallelism allows computing std and mean in parallel. So if we parallelize them by operator dimension into 2 
devices (cuda:0, cuda:1), first we copy input data into both devices, and cuda:0 computes std, cuda:1 computes mean at the same time.

* Attribute

We have 10 batches of 512 length. If we parallelize them by attribute dimension into 2 devices, 10 x 512 will be 10 x 2 x 256.

* Parameter

It is similar with tensor model parallelism or naive layer-wise model parallelism.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-flexflow.jpeg" alt="flex-flow-soap"/>
</div>

The significance of this framework is that it takes resources like (1) GPU/TPU/CPU vs. (2) RAM/DRAM vs. (3) 
fast-intra-connect/slow-inter-connect and it automatically optimizes all these algorithmically deciding which 
parallelisation to use where.

One very important aspect is that FlexFlow is designed for optimizing DNN parallelizations for models with static and 
fixed workloads, since models with dynamic behavior may prefer different parallelization strategies across iterations.

So the promise is very attractive - it runs a 30min simulation on the cluster of choice and it comes up with the best 
strategy to utilise this specific environment. If you add/remove/replace any parts it'll run and re-optimize the plan 
for that. And then you can train. A different setup will have its own custom optimization.

🤗 Transformers status: Transformers models are FX-trace-able via [transformers.utils.fx](https://github.com/huggingface/transformers/blob/master/src/transformers/utils/fx.py), 
which is a prerequisite for FlexFlow, however, changes are required on the FlexFlow side to make it work with Transformers models.

## GPU selection

When training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first.


key: _node_content
string_value:
{"id_": "4b8a40a0-d996-4cfe-ba92-0287b2bd216d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "65dd47e0-ebd5-497b-92ec-6c46f2c48ae2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "36e87959957cbc663eb512609fae1a0a1bdd8589b93c75eaf3a0cff0bc8a1589", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "77c2c817-1e29-4c24-887b-d0c5b4f6457b", "node_type": "1", "metadata": {}, "hash": "ec9e668809547ed02db8f27a69bfa8e817b496b9f5b05eadebe77e201862218f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31245, "end_char_idx": 34248, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If we parallelize them by attribute dimension into 2 devices, 10 x 512 will be 10 x 2 x 256.

* Parameter

It is similar with tensor model parallelism or naive layer-wise model parallelism.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-flexflow.jpeg" alt="flex-flow-soap"/>
</div>

The significance of this framework is that it takes resources like (1) GPU/TPU/CPU vs. (2) RAM/DRAM vs. (3) 
fast-intra-connect/slow-inter-connect and it automatically optimizes all these algorithmically deciding which 
parallelisation to use where.

One very important aspect is that FlexFlow is designed for optimizing DNN parallelizations for models with static and 
fixed workloads, since models with dynamic behavior may prefer different parallelization strategies across iterations.

So the promise is very attractive - it runs a 30min simulation on the cluster of choice and it comes up with the best 
strategy to utilise this specific environment. If you add/remove/replace any parts it'll run and re-optimize the plan 
for that. And then you can train. A different setup will have its own custom optimization.

🤗 Transformers status: Transformers models are FX-trace-able via [transformers.utils.fx](https://github.com/huggingface/transformers/blob/master/src/transformers/utils/fx.py), 
which is a prerequisite for FlexFlow, however, changes are required on the FlexFlow side to make it work with Transformers models.

## GPU selection

When training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first. The selection process works for both [DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) and [DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html) to use only a subset of the available GPUs, and you don't need Accelerate or the [DeepSpeed integration](./main_classes/deepspeed).

### Number of GPUs

For example, if you have 4 GPUs and you only want to use the first 2:

<hfoptions id="select-gpu">
<hfoption id="torchrun">

Use the `--nproc_per_node` to select how many GPUs to use.

```bash
torchrun --nproc_per_node=2  trainer-program.py ...
```

</hfoption>
<hfoption id="Accelerate">

Use `--num_processes` to select how many GPUs to use.

```bash
accelerate launch --num_processes 2 trainer-program.py ...
```

</hfoption>
<hfoption id="DeepSpeed">

Use `--num_gpus` to select how many GPUs to use.

```bash
deepspeed --num_gpus 2 trainer-program.py ...
```

</hfoption>
</hfoptions>

### Order of GPUs

Now, to select which GPUs to use and their order, you'll use the `CUDA_VISIBLE_DEVICES` environment variable. It is easiest to set the environment variable in a `~/bashrc` or another startup config file. `CUDA_VISIBLE_DEVICES` is used to map which GPUs are used.


key: _node_content
string_value:
{"id_": "77c2c817-1e29-4c24-887b-d0c5b4f6457b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b8a40a0-d996-4cfe-ba92-0287b2bd216d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "286fff1222eeb809215f173368f63d99883d85ab23e65ea2f9f517adb6a25428", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "15877862-4fdd-4907-ae1c-9615827483fe", "node_type": "1", "metadata": {}, "hash": "d9887dcaa861f52c59264cf00383a0d11b00c25a3b1f749e63a7580a63931f94", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32750, "end_char_idx": 35762, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## GPU selection

When training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first. The selection process works for both [DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) and [DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html) to use only a subset of the available GPUs, and you don't need Accelerate or the [DeepSpeed integration](./main_classes/deepspeed).

### Number of GPUs

For example, if you have 4 GPUs and you only want to use the first 2:

<hfoptions id="select-gpu">
<hfoption id="torchrun">

Use the `--nproc_per_node` to select how many GPUs to use.

```bash
torchrun --nproc_per_node=2  trainer-program.py ...
```

</hfoption>
<hfoption id="Accelerate">

Use `--num_processes` to select how many GPUs to use.

```bash
accelerate launch --num_processes 2 trainer-program.py ...
```

</hfoption>
<hfoption id="DeepSpeed">

Use `--num_gpus` to select how many GPUs to use.

```bash
deepspeed --num_gpus 2 trainer-program.py ...
```

</hfoption>
</hfoptions>

### Order of GPUs

Now, to select which GPUs to use and their order, you'll use the `CUDA_VISIBLE_DEVICES` environment variable. It is easiest to set the environment variable in a `~/bashrc` or another startup config file. `CUDA_VISIBLE_DEVICES` is used to map which GPUs are used. For example, if you have 4 GPUs (0, 1, 2, 3) and you only want to run GPUs 0 and 2:

```bash
CUDA_VISIBLE_DEVICES=0,2 torchrun trainer-program.py ...
```

Only the 2 physical GPUs (0 and 2) are "visible" to PyTorch and these are mapped to `cuda:0` and `cuda:1` respectively. You can also reverse the order of the GPUs to use 2 first. Now, the mapping is `cuda:1` for GPU 0 and `cuda:0` for GPU 2.

```bash
CUDA_VISIBLE_DEVICES=2,0 torchrun trainer-program.py ...
```

You can also set the `CUDA_VISIBLE_DEVICES` environment variable to an empty value to create an environment without GPUs.

```bash
CUDA_VISIBLE_DEVICES= python trainer-program.py ...
```

<Tip warning={true}>

As with any environment variable, they can be exported instead of being added to the command line. However, this is not recommended because it can be confusing if you forget how the environment variable was setup and you end up using the wrong GPUs. Instead, it is common practice to set the environment variable for a specific training run on the same command line.

</Tip>

`CUDA_DEVICE_ORDER` is an alternative environment variable you can use to control how the GPUs are ordered. You can either order them by:

1. PCIe bus ID's that matches the order of [`nvidia-smi`](https://developer.nvidia.com/nvidia-system-management-interface) and [`rocm-smi`](https://rocm.docs.amd.com/projects/rocm_smi_lib/en/latest/.doxygen/docBin/html/index.html) for NVIDIA and AMD GPUs respectively

```bash
export CUDA_DEVICE_ORDER=PCI_BUS_ID
```

2.


key: _node_content
string_value:
{"id_": "15877862-4fdd-4907-ae1c-9615827483fe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75b419f5-67bc-472c-959d-d30c5e65d515", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "77c2c817-1e29-4c24-887b-d0c5b4f6457b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "f19c9d3c93d37e938a5569698ab3b47d1084f3f9c6ba8910b2247aee442c86f2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34249, "end_char_idx": 36278, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, if you have 4 GPUs (0, 1, 2, 3) and you only want to run GPUs 0 and 2:

```bash
CUDA_VISIBLE_DEVICES=0,2 torchrun trainer-program.py ...
```

Only the 2 physical GPUs (0 and 2) are "visible" to PyTorch and these are mapped to `cuda:0` and `cuda:1` respectively. You can also reverse the order of the GPUs to use 2 first. Now, the mapping is `cuda:1` for GPU 0 and `cuda:0` for GPU 2.

```bash
CUDA_VISIBLE_DEVICES=2,0 torchrun trainer-program.py ...
```

You can also set the `CUDA_VISIBLE_DEVICES` environment variable to an empty value to create an environment without GPUs.

```bash
CUDA_VISIBLE_DEVICES= python trainer-program.py ...
```

<Tip warning={true}>

As with any environment variable, they can be exported instead of being added to the command line. However, this is not recommended because it can be confusing if you forget how the environment variable was setup and you end up using the wrong GPUs. Instead, it is common practice to set the environment variable for a specific training run on the same command line.

</Tip>

`CUDA_DEVICE_ORDER` is an alternative environment variable you can use to control how the GPUs are ordered. You can either order them by:

1. PCIe bus ID's that matches the order of [`nvidia-smi`](https://developer.nvidia.com/nvidia-system-management-interface) and [`rocm-smi`](https://rocm.docs.amd.com/projects/rocm_smi_lib/en/latest/.doxygen/docBin/html/index.html) for NVIDIA and AMD GPUs respectively

```bash
export CUDA_DEVICE_ORDER=PCI_BUS_ID
```

2. GPU compute ability

```bash
export CUDA_DEVICE_ORDER=FASTEST_FIRST
```

The `CUDA_DEVICE_ORDER` is especially useful if your training setup consists of an older and newer GPU, where the older GPU appears first, but you cannot physically swap the cards to make the newer GPU appear first. In this case, set `CUDA_DEVICE_ORDER=FASTEST_FIRST` to always use the newer and faster GPU first (`nvidia-smi` or `rocm-smi` still reports the GPUs in their PCIe order). Or you could also set `export CUDA_VISIBLE_DEVICES=1,0`.


key: _node_content
string_value:
{"id_": "099f5f71-4cc6-454b-be5d-a43083857d6f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "87d62b21-1e33-4aeb-8fd2-a9c71deed06d", "node_type": "1", "metadata": {}, "hash": "aaf2f9718a6a3c761152d24cd7272130958591333fb5cb963ccc19c90d0ddb3b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Chat Templates

## Introduction

An increasingly common use case for LLMs is **chat**. In a chat context, rather than continuing a single string
of text (as is the case with a standard language model), the model instead continues a conversation that consists
of one or more **messages**, each of which includes a **role**, like "user" or "assistant", as well as message text.

Much like tokenization, different models expect very different input formats for chat. This is the reason we added
**chat templates** as a feature. Chat templates are part of the tokenizer. They specify how to convert conversations, 
represented as lists of messages, into a single tokenizable string in the format that the model expects. 

Let's make this concrete with a quick example using the `BlenderBot` model. BlenderBot has an extremely simple default 
template, which mostly just adds whitespace between rounds of dialogue:

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")

>>> chat = [
...    {"role": "user", "content": "Hello, how are you?"},
...    {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
...    {"role": "user", "content": "I'd like to show off how chat templating works!"},
... ]

>>> tokenizer.apply_chat_template(chat, tokenize=False)
" Hello, how are you?  I'm doing great. How can I help you today?   I'd like to show off how chat templating works!</s>"
```

Notice how the entire chat is condensed into a single string. If we use `tokenize=True`, which is the default setting,
that string will also be tokenized for us. To see a more complex template in action, though, let's use the 
`mistralai/Mistral-7B-Instruct-v0.1` model.

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")

>>> chat = [
...   {"role": "user", "content": "Hello, how are you?"},
...   {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
...   {"role": "user", "content": "I'd like to show off how chat templating works!"},
... ]

>>> tokenizer.apply_chat_template(chat, tokenize=False)
"<s>[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]"
```

Note that this time, the tokenizer has added the control tokens [INST] and [/INST] to indicate the start and end of 
user messages (but not assistant messages!). Mistral-instruct was trained with these tokens, but BlenderBot was not.

## How do I use chat templates?

As you can see in the example above, chat templates are easy to use. Simply build a list of messages, with `role`
and `content` keys, and then pass it to the [`~PreTrainedTokenizer.apply_chat_template`] method. Once you do that,
you'll get output that's ready to go! When using chat templates as input for model generation, it's also a good idea
to use `add_generation_prompt=True` to add a [generation prompt](#what-are-generation-prompts).


key: _node_content
string_value:
{"id_": "87d62b21-1e33-4aeb-8fd2-a9c71deed06d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "099f5f71-4cc6-454b-be5d-a43083857d6f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "eaf497683821b874f441589bb2054bc2912e984a52d28fb30a67b26047496e6f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "28c17bfb-2414-4f5e-b180-2405e531de8c", "node_type": "1", "metadata": {}, "hash": "e25898941bcd9e254bdf0854b82d2f2b42af8802a836f50eca862bc3e83d739b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1646, "end_char_idx": 4656, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To see a more complex template in action, though, let's use the 
`mistralai/Mistral-7B-Instruct-v0.1` model.

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")

>>> chat = [
...   {"role": "user", "content": "Hello, how are you?"},
...   {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
...   {"role": "user", "content": "I'd like to show off how chat templating works!"},
... ]

>>> tokenizer.apply_chat_template(chat, tokenize=False)
"<s>[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]"
```

Note that this time, the tokenizer has added the control tokens [INST] and [/INST] to indicate the start and end of 
user messages (but not assistant messages!). Mistral-instruct was trained with these tokens, but BlenderBot was not.

## How do I use chat templates?

As you can see in the example above, chat templates are easy to use. Simply build a list of messages, with `role`
and `content` keys, and then pass it to the [`~PreTrainedTokenizer.apply_chat_template`] method. Once you do that,
you'll get output that's ready to go! When using chat templates as input for model generation, it's also a good idea
to use `add_generation_prompt=True` to add a [generation prompt](#what-are-generation-prompts). 

Here's an example of preparing input for `model.generate()`, using the `Zephyr` assistant model:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "HuggingFaceH4/zephyr-7b-beta"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)  # You may want to use bfloat16 and/or move to GPU here

messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
 ]
tokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
print(tokenizer.decode(tokenized_chat[0]))
```
This will yield a string in the input format that Zephyr expects. 
```text
<|system|>
You are a friendly chatbot who always responds in the style of a pirate</s> 
<|user|>
How many helicopters can a human eat in one sitting?</s> 
<|assistant|>
```

Now that our input is formatted correctly for Zephyr, we can use the model to generate a response to the user's question:

```python
outputs = model.generate(tokenized_chat, max_new_tokens=128) 
print(tokenizer.decode(outputs[0]))
```

This will yield:

```text
<|system|>
You are a friendly chatbot who always responds in the style of a pirate</s> 
<|user|>
How many helicopters can a human eat in one sitting?</s> 
<|assistant|>
Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines.


key: _node_content
string_value:
{"id_": "28c17bfb-2414-4f5e-b180-2405e531de8c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "87d62b21-1e33-4aeb-8fd2-a9c71deed06d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "43ac11649d7b1b51fd1c999211a244b9f4519f6531e0cfea6b7730867854c1fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad11a581-2675-46da-b73e-62dd6179e60b", "node_type": "1", "metadata": {}, "hash": "f9f544acf6838481cccf8e0cb3f215a2b7dd1199c83fd4f9a26b4699fe3f0e14", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3066, "end_char_idx": 6128, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here's an example of preparing input for `model.generate()`, using the `Zephyr` assistant model:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "HuggingFaceH4/zephyr-7b-beta"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)  # You may want to use bfloat16 and/or move to GPU here

messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
 ]
tokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
print(tokenizer.decode(tokenized_chat[0]))
```
This will yield a string in the input format that Zephyr expects. 
```text
<|system|>
You are a friendly chatbot who always responds in the style of a pirate</s> 
<|user|>
How many helicopters can a human eat in one sitting?</s> 
<|assistant|>
```

Now that our input is formatted correctly for Zephyr, we can use the model to generate a response to the user's question:

```python
outputs = model.generate(tokenized_chat, max_new_tokens=128) 
print(tokenizer.decode(outputs[0]))
```

This will yield:

```text
<|system|>
You are a friendly chatbot who always responds in the style of a pirate</s> 
<|user|>
How many helicopters can a human eat in one sitting?</s> 
<|assistant|>
Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all.
```

Arr, 'twas easy after all!

## Is there an automated pipeline for chat?

Yes, there is! Our text generation pipelines support chat inputs, which makes it easy to use chat models. In the past,
we used to use a dedicated "ConversationalPipeline" class, but this has now been deprecated and its functionality
has been merged into the [`TextGenerationPipeline`]. Let's try the `Zephyr` example again, but this time using 
a pipeline:

```python
from transformers import pipeline

pipe = pipeline("text-generation", "HuggingFaceH4/zephyr-7b-beta")
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
print(pipe(messages, max_new_tokens=128)[0]['generated_text'][-1])  # Print the assistant's response
```

```text
{'role': 'assistant', 'content': "Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'.


key: _node_content
string_value:
{"id_": "ad11a581-2675-46da-b73e-62dd6179e60b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "28c17bfb-2414-4f5e-b180-2405e531de8c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "89348a1f8bcd6c56dcbdc949ffc9eb9867999ced771d268809ce12e1ea98be1a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3e27a075-f85f-461b-9e20-b17b7a82bd5f", "node_type": "1", "metadata": {}, "hash": "b7f27af8fe5a32cfaa70506fe28a9d578d588fb697a2d7c25045332f6db5780b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4605, "end_char_idx": 6856, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all.
```

Arr, 'twas easy after all!

## Is there an automated pipeline for chat?

Yes, there is! Our text generation pipelines support chat inputs, which makes it easy to use chat models. In the past,
we used to use a dedicated "ConversationalPipeline" class, but this has now been deprecated and its functionality
has been merged into the [`TextGenerationPipeline`]. Let's try the `Zephyr` example again, but this time using 
a pipeline:

```python
from transformers import pipeline

pipe = pipeline("text-generation", "HuggingFaceH4/zephyr-7b-beta")
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
print(pipe(messages, max_new_tokens=128)[0]['generated_text'][-1])  # Print the assistant's response
```

```text
{'role': 'assistant', 'content': "Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all."}
```

The pipeline will take care of all the details of tokenization and calling `apply_chat_template` for you -
once the model has a chat template, all you need to do is initialize the pipeline and pass it the list of messages!

## What are "generation prompts"?

You may have noticed that the `apply_chat_template` method has an `add_generation_prompt` argument. This argument tells
the template to add tokens that indicate the start of a bot response. For example, consider the following chat:

```python
messages = [
    {"role": "user", "content": "Hi there!"},
    {"role": "assistant", "content": "Nice to meet you!"},
    {"role": "user", "content": "Can I ask a question?"}


key: _node_content
string_value:
{"id_": "3e27a075-f85f-461b-9e20-b17b7a82bd5f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad11a581-2675-46da-b73e-62dd6179e60b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "d08afdbaac7f024773418d961b4ee32ce17b76cb3788645220cb9f6460909a56", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bf38b199-892d-416b-af40-392b0fe419ec", "node_type": "1", "metadata": {}, "hash": "b7cba7e31165069a5c93390b842c3e08ed0287ee8aeaa932b957c198d3f3ecc9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5666, "end_char_idx": 8818, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


