# embedding_metadata

key: _node_content
string_value:
{"id_": "ef4a466c-9ffa-4d90-bd1f-308dabd121ac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9222a9f9-7666-49c0-ab16-8394389abd03", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers"}, "hash": "4080bf02bd42c08da311b192ac1c7aa133ecbf6a8a68562c9f70249f67176977", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2691a3e4-32a3-4cd7-8367-9b6ba3fa409e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/performance", "title": "Performance and Scalability", "tokens": 838, "retrieve_doc": true, "source": "transformers"}, "hash": "a0b9f20f17aacdbe8c4880837040b0b356989bde3873ad7a6d6fd883483d5b79", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3355, "end_char_idx": 4029, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Training and inference

Here you'll find techniques, tips and tricks that apply whether you are training a model, or running inference with it.

* [Instantiating a big model](big_models)
* [Troubleshooting performance issues](debugging)

## Contribute

This document is far from being complete and a lot more needs to be added, so if you have additions or corrections to 
make please don't hesitate to open a PR or if you aren't sure start an Issue and we can discuss the details there.

When making contributions that A is better than B, please try to include a reproducible benchmark and/or a link to the 
source of that information (unless it comes directly from you).


key: _node_content
string_value:
{"id_": "d01c7baa-83d9-41c6-938c-7263a2eebe66", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ff5a904-2dd0-42c3-ab52-3cc4a5c03cf7", "node_type": "1", "metadata": {}, "hash": "86b9274f60676ae660c2b6f1e47b78707b5ff3719e8f22033304d757e3dc5e0b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3580, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# What 🤗 Transformers can do

🤗 Transformers is a library of pretrained state-of-the-art models for natural language processing (NLP), computer vision, and audio and speech processing tasks. Not only does the library contain Transformer models, but it also has non-Transformer models like modern convolutional networks for computer vision tasks. If you look at some of the most popular consumer products today, like smartphones, apps, and televisions, odds are that some kind of deep learning technology is behind it. Want to remove a background object from a picture taken by your smartphone? This is an example of a panoptic segmentation task (don't worry if you don't know what this means yet, we'll describe it in the following sections!). 

This page provides an overview of the different speech and audio, computer vision, and NLP tasks that can be solved with the 🤗 Transformers library in just three lines of code!

## Audio

Audio and speech processing tasks are a little different from the other modalities mainly because audio as an input is a continuous signal. Unlike text, a raw audio waveform can't be neatly split into discrete chunks the way a sentence can be divided into words. To get around this, the raw audio signal is typically sampled at regular intervals. If you take more samples within an interval, the sampling rate is higher, and the audio more closely resembles the original audio source.

Previous approaches preprocessed the audio to extract useful features from it. It is now more common to start audio and speech processing tasks by directly feeding the raw audio waveform to a feature encoder to extract an audio representation. This simplifies the preprocessing step and allows the model to learn the most essential features.

### Audio classification

Audio classification is a task that labels audio data from a predefined set of classes. It is a broad category with many specific applications, some of which include:

* acoustic scene classification: label audio with a scene label ("office", "beach", "stadium")
* acoustic event detection: label audio with a sound event label ("car horn", "whale calling", "glass breaking")
* tagging: label audio containing multiple sounds (birdsongs, speaker identification in a meeting)
* music classification: label music with a genre label ("metal", "hip-hop", "country")

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="audio-classification", model="superb/hubert-base-superb-er")
>>> preds = classifier("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.4532, 'label': 'hap'},
 {'score': 0.3622, 'label': 'sad'},
 {'score': 0.0943, 'label': 'neu'},
 {'score': 0.0903, 'label': 'ang'}]
```

### Automatic speech recognition

Automatic speech recognition (ASR) transcribes speech into text. It is one of the most common audio tasks due partly to speech being such a natural form of human communication. Today, ASR systems are embedded in "smart" technology products like speakers, phones, and cars. We can ask our virtual assistants to play music, set reminders, and tell us the weather. 

But one of the key challenges Transformer architectures have helped with is in low-resource languages. By pretraining on large amounts of speech data, finetuning the model on only one hour of labeled speech data in a low-resource language can still produce high-quality results compared to previous ASR systems trained on 100x more labeled data.


key: _node_content
string_value:
{"id_": "4ff5a904-2dd0-42c3-ab52-3cc4a5c03cf7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d01c7baa-83d9-41c6-938c-7263a2eebe66", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "25eb3f029331abd1f689c5175e8884a2c8c5e1defc3ef469088cccf2fd86076a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c864edd4-68f8-4d85-a3ba-775c8aacc867", "node_type": "1", "metadata": {}, "hash": "4217f64a15a9898c2b855905d6c1a845687f16da3f284c4416b095abb8789ae4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2936, "end_char_idx": 6277, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It is one of the most common audio tasks due partly to speech being such a natural form of human communication. Today, ASR systems are embedded in "smart" technology products like speakers, phones, and cars. We can ask our virtual assistants to play music, set reminders, and tell us the weather. 

But one of the key challenges Transformer architectures have helped with is in low-resource languages. By pretraining on large amounts of speech data, finetuning the model on only one hour of labeled speech data in a low-resource language can still produce high-quality results compared to previous ASR systems trained on 100x more labeled data.

```py
>>> from transformers import pipeline

>>> transcriber = pipeline(task="automatic-speech-recognition", model="openai/whisper-small")
>>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
{'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'}
```

## Computer vision

One of the first and earliest successful computer vision tasks was recognizing images of zip code numbers using a [convolutional neural network (CNN)](glossary#convolution). An image is composed of pixels, and each pixel has a numerical value. This makes it easy to represent an image as a matrix of pixel values. Each particular combination of pixel values describes the colors of an image. 

Two general ways computer vision tasks can be solved are:

1. Use convolutions to learn the hierarchical features of an image from low-level features to high-level abstract things.
2. Split an image into patches and use a Transformer to gradually learn how each image patch is related to each other to form an image. Unlike the bottom-up approach favored by a CNN, this is kind of like starting out with a blurry image and then gradually bringing it into focus.

### Image classification

Image classification labels an entire image from a predefined set of classes. Like most classification tasks, there are many practical use cases for image classification, some of which include:

* healthcare: label medical images to detect disease or monitor patient health
* environment: label satellite images to monitor deforestation, inform wildland management or detect wildfires
* agriculture: label images of crops to monitor plant health or satellite images for land use monitoring 
* ecology: label images of animal or plant species to monitor wildlife populations or track endangered species

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="image-classification")
>>> preds = classifier(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.4335, 'label': 'lynx, catamount'}
{'score': 0.0348, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}
{'score': 0.0324, 'label': 'snow leopard, ounce, Panthera uncia'}
{'score': 0.0239, 'label': 'Egyptian cat'}
{'score': 0.0229, 'label': 'tiger cat'}
```

### Object detection

Unlike image classification, object detection identifies multiple objects within an image and the objects' positions in an image (defined by the bounding box).


key: _node_content
string_value:
{"id_": "c864edd4-68f8-4d85-a3ba-775c8aacc867", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4ff5a904-2dd0-42c3-ab52-3cc4a5c03cf7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "e3b26371b7397c4bed0c6457ab0a936cf202e2e15addb338512e81f6f599633b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b61feaf2-fcde-410f-ad2f-c0e1d3a69dd1", "node_type": "1", "metadata": {}, "hash": "cb7651ddc995abe45051847dfddef6855e5f108001faa70a06f9a62d6d23115e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4524, "end_char_idx": 8045, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Split an image into patches and use a Transformer to gradually learn how each image patch is related to each other to form an image. Unlike the bottom-up approach favored by a CNN, this is kind of like starting out with a blurry image and then gradually bringing it into focus.

### Image classification

Image classification labels an entire image from a predefined set of classes. Like most classification tasks, there are many practical use cases for image classification, some of which include:

* healthcare: label medical images to detect disease or monitor patient health
* environment: label satellite images to monitor deforestation, inform wildland management or detect wildfires
* agriculture: label images of crops to monitor plant health or satellite images for land use monitoring 
* ecology: label images of animal or plant species to monitor wildlife populations or track endangered species

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="image-classification")
>>> preds = classifier(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.4335, 'label': 'lynx, catamount'}
{'score': 0.0348, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}
{'score': 0.0324, 'label': 'snow leopard, ounce, Panthera uncia'}
{'score': 0.0239, 'label': 'Egyptian cat'}
{'score': 0.0229, 'label': 'tiger cat'}
```

### Object detection

Unlike image classification, object detection identifies multiple objects within an image and the objects' positions in an image (defined by the bounding box). Some example applications of object detection include:

* self-driving vehicles: detect everyday traffic objects such as other vehicles, pedestrians, and traffic lights
* remote sensing: disaster monitoring, urban planning, and weather forecasting
* defect detection: detect cracks or structural damage in buildings, and manufacturing defects

```py
>>> from transformers import pipeline

>>> detector = pipeline(task="object-detection")
>>> preds = detector(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"], "box": pred["box"]} for pred in preds]
>>> preds
[{'score': 0.9865,
  'label': 'cat',
  'box': {'xmin': 178, 'ymin': 154, 'xmax': 882, 'ymax': 598}}]
```

### Image segmentation

Image segmentation is a pixel-level task that assigns every pixel in an image to a class. It differs from object detection, which uses bounding boxes to label and predict objects in an image because segmentation is more granular. Segmentation can detect objects at a pixel-level. There are several types of image segmentation:

* instance segmentation: in addition to labeling the class of an object, it also labels each distinct instance of an object ("dog-1", "dog-2")
* panoptic segmentation: a combination of semantic and instance segmentation; it labels each pixel with a semantic class **and** each distinct instance of an object

Segmentation tasks are helpful in self-driving vehicles to create a pixel-level map of the world around them so they can navigate safely around pedestrians and other vehicles. It is also useful for medical imaging, where the task's finer granularity can help identify abnormal cells or organ features.


key: _node_content
string_value:
{"id_": "b61feaf2-fcde-410f-ad2f-c0e1d3a69dd1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c864edd4-68f8-4d85-a3ba-775c8aacc867", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "54a2f4ef00d609ba4184951458019bc5c3d5d1e549af25fe7046e5ec03c6c544", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "05f25b90-8395-427d-b2fc-1feb309299b5", "node_type": "1", "metadata": {}, "hash": "ee65d4d906f42f271bce8e33034bbd9f2632a73029434b0e9788e112ba4c8957", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6278, "end_char_idx": 9846, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Some example applications of object detection include:

* self-driving vehicles: detect everyday traffic objects such as other vehicles, pedestrians, and traffic lights
* remote sensing: disaster monitoring, urban planning, and weather forecasting
* defect detection: detect cracks or structural damage in buildings, and manufacturing defects

```py
>>> from transformers import pipeline

>>> detector = pipeline(task="object-detection")
>>> preds = detector(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"], "box": pred["box"]} for pred in preds]
>>> preds
[{'score': 0.9865,
  'label': 'cat',
  'box': {'xmin': 178, 'ymin': 154, 'xmax': 882, 'ymax': 598}}]
```

### Image segmentation

Image segmentation is a pixel-level task that assigns every pixel in an image to a class. It differs from object detection, which uses bounding boxes to label and predict objects in an image because segmentation is more granular. Segmentation can detect objects at a pixel-level. There are several types of image segmentation:

* instance segmentation: in addition to labeling the class of an object, it also labels each distinct instance of an object ("dog-1", "dog-2")
* panoptic segmentation: a combination of semantic and instance segmentation; it labels each pixel with a semantic class **and** each distinct instance of an object

Segmentation tasks are helpful in self-driving vehicles to create a pixel-level map of the world around them so they can navigate safely around pedestrians and other vehicles. It is also useful for medical imaging, where the task's finer granularity can help identify abnormal cells or organ features. Image segmentation can also be used in ecommerce to virtually try on clothes or create augmented reality experiences by overlaying objects in the real world through your camera.

```py
>>> from transformers import pipeline

>>> segmenter = pipeline(task="image-segmentation")
>>> preds = segmenter(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.9879, 'label': 'LABEL_184'}
{'score': 0.9973, 'label': 'snow'}
{'score': 0.9972, 'label': 'cat'}
```

### Depth estimation

Depth estimation predicts the distance of each pixel in an image from the camera. This computer vision task is especially important for scene understanding and reconstruction. For example, in self-driving cars, vehicles need to understand how far objects like pedestrians, traffic signs, and other vehicles are to avoid obstacles and collisions. Depth information is also helpful for constructing 3D representations from 2D images and can be used to create high-quality 3D representations of biological structures or buildings.

There are two approaches to depth estimation:

* stereo: depths are estimated by comparing two images of the same image from slightly different angles
* monocular: depths are estimated from a single image

```py
>>> from transformers import pipeline

>>> depth_estimator = pipeline(task="depth-estimation")
>>> preds = depth_estimator(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
```

## Natural language processing

NLP tasks are among the most common types of tasks because text is such a natural way for us to communicate.


key: _node_content
string_value:
{"id_": "05f25b90-8395-427d-b2fc-1feb309299b5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b61feaf2-fcde-410f-ad2f-c0e1d3a69dd1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "bf41c7dba5e3caa636cd4f5c375c68a0d98dc973c60b19ca9337fa1bf6a385a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "461ba8a0-4da8-49b2-bff0-0af11ca65434", "node_type": "1", "metadata": {}, "hash": "f01aa8c49f0bbe17fa282f2a40693c8394b6bf925081bc63da7356def6f64d05", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8046, "end_char_idx": 11630, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Image segmentation can also be used in ecommerce to virtually try on clothes or create augmented reality experiences by overlaying objects in the real world through your camera.

```py
>>> from transformers import pipeline

>>> segmenter = pipeline(task="image-segmentation")
>>> preds = segmenter(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.9879, 'label': 'LABEL_184'}
{'score': 0.9973, 'label': 'snow'}
{'score': 0.9972, 'label': 'cat'}
```

### Depth estimation

Depth estimation predicts the distance of each pixel in an image from the camera. This computer vision task is especially important for scene understanding and reconstruction. For example, in self-driving cars, vehicles need to understand how far objects like pedestrians, traffic signs, and other vehicles are to avoid obstacles and collisions. Depth information is also helpful for constructing 3D representations from 2D images and can be used to create high-quality 3D representations of biological structures or buildings.

There are two approaches to depth estimation:

* stereo: depths are estimated by comparing two images of the same image from slightly different angles
* monocular: depths are estimated from a single image

```py
>>> from transformers import pipeline

>>> depth_estimator = pipeline(task="depth-estimation")
>>> preds = depth_estimator(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
```

## Natural language processing

NLP tasks are among the most common types of tasks because text is such a natural way for us to communicate. To get text into a format recognized by a model, it needs to be tokenized. This means dividing a sequence of text into separate words or subwords (tokens) and then converting these tokens into numbers. As a result, you can represent a sequence of text as a sequence of numbers, and once you have a sequence of numbers, it can be input into a model to solve all sorts of NLP tasks!

### Text classification

Like classification tasks in any modality, text classification labels a sequence of text (it can be sentence-level, a paragraph, or a document) from a predefined set of classes. There are many practical applications for text classification, some of which include:

* sentiment analysis: label text according to some polarity like `positive` or `negative` which can inform and support decision-making in fields like politics, finance, and marketing
* content classification: label text according to some topic to help organize and filter information in news and social media feeds (`weather`, `sports`, `finance`, etc.)

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="sentiment-analysis")
>>> preds = classifier("Hugging Face is the best thing since sliced bread!")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.9991, 'label': 'POSITIVE'}]
```

### Token classification

In any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as [tokens](glossary#token). Token classification assigns each token a label from a predefined set of classes. 

Two common types of token classification are:

* named entity recognition (NER): label a token according to an entity category like organization, person, location or date.


key: _node_content
string_value:
{"id_": "461ba8a0-4da8-49b2-bff0-0af11ca65434", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "05f25b90-8395-427d-b2fc-1feb309299b5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "527efa3019b5c9c54efa55459975989184db1f2eea3ae00592a690b29705566e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96c126ee-bba7-4229-8e80-bc01423634a0", "node_type": "1", "metadata": {}, "hash": "dacb7c0bad5c895b8c6d63b6947868cef3eb2e897f90ce9e2402c4d66d752648", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9847, "end_char_idx": 12157, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To get text into a format recognized by a model, it needs to be tokenized. This means dividing a sequence of text into separate words or subwords (tokens) and then converting these tokens into numbers. As a result, you can represent a sequence of text as a sequence of numbers, and once you have a sequence of numbers, it can be input into a model to solve all sorts of NLP tasks!

### Text classification

Like classification tasks in any modality, text classification labels a sequence of text (it can be sentence-level, a paragraph, or a document) from a predefined set of classes. There are many practical applications for text classification, some of which include:

* sentiment analysis: label text according to some polarity like `positive` or `negative` which can inform and support decision-making in fields like politics, finance, and marketing
* content classification: label text according to some topic to help organize and filter information in news and social media feeds (`weather`, `sports`, `finance`, etc.)

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="sentiment-analysis")
>>> preds = classifier("Hugging Face is the best thing since sliced bread!")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.9991, 'label': 'POSITIVE'}]
```

### Token classification

In any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as [tokens](glossary#token). Token classification assigns each token a label from a predefined set of classes. 

Two common types of token classification are:

* named entity recognition (NER): label a token according to an entity category like organization, person, location or date. NER is especially popular in biomedical settings, where it can label genes, proteins, and drug names.
* part-of-speech tagging (POS): label a token according to its part-of-speech like noun, verb, or adjective. POS is useful for helping translation systems understand how two identical words are grammatically different (bank as a noun versus bank as a verb).

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="ner")
>>> preds = classifier("Hugging Face is a French company based in New York City.")


key: _node_content
string_value:
{"id_": "96c126ee-bba7-4229-8e80-bc01423634a0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "461ba8a0-4da8-49b2-bff0-0af11ca65434", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "970d1e6e4dbda2b42cb19bfa0f039b5b6aefa50d500204ddba357a0ba50e0380", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5d1ce008-57a2-4902-a770-5915e8eeb5c8", "node_type": "1", "metadata": {}, "hash": "f29c07dad20ae21722adf83d6536ae294fb99f95b75a4a1d9e83061eaf2b83ec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10432, "end_char_idx": 13296, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
There are many practical applications for text classification, some of which include:

* sentiment analysis: label text according to some polarity like `positive` or `negative` which can inform and support decision-making in fields like politics, finance, and marketing
* content classification: label text according to some topic to help organize and filter information in news and social media feeds (`weather`, `sports`, `finance`, etc.)

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="sentiment-analysis")
>>> preds = classifier("Hugging Face is the best thing since sliced bread!")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.9991, 'label': 'POSITIVE'}]
```

### Token classification

In any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as [tokens](glossary#token). Token classification assigns each token a label from a predefined set of classes. 

Two common types of token classification are:

* named entity recognition (NER): label a token according to an entity category like organization, person, location or date. NER is especially popular in biomedical settings, where it can label genes, proteins, and drug names.
* part-of-speech tagging (POS): label a token according to its part-of-speech like noun, verb, or adjective. POS is useful for helping translation systems understand how two identical words are grammatically different (bank as a noun versus bank as a verb).

```py
>>> from transformers import pipeline

>>> classifier = pipeline(task="ner")
>>> preds = classifier("Hugging Face is a French company based in New York City.")
>>> preds = [
...     {
...         "entity": pred["entity"],
...         "score": round(pred["score"], 4),
...         "index": pred["index"],
...         "word": pred["word"],
...         "start": pred["start"],
...         "end": pred["end"],
...     }
...     for pred in preds
... ]
>>> print(*preds, sep="\n")
{'entity': 'I-ORG', 'score': 0.9968, 'index': 1, 'word': 'Hu', 'start': 0, 'end': 2}
{'entity': 'I-ORG', 'score': 0.9293, 'index': 2, 'word': '##gging', 'start': 2, 'end': 7}
{'entity': 'I-ORG', 'score': 0.9763, 'index': 3, 'word': 'Face', 'start': 8, 'end': 12}
{'entity': 'I-MISC', 'score': 0.9983, 'index': 6, 'word': 'French', 'start': 18, 'end': 24}
{'entity': 'I-LOC', 'score': 0.999, 'index': 10, 'word': 'New', 'start': 42, 'end': 45}
{'entity': 'I-LOC', 'score': 0.9987, 'index': 11, 'word': 'York', 'start': 46, 'end': 50}
{'entity': 'I-LOC', 'score': 0.9992, 'index': 12, 'word': 'City', 'start': 51, 'end': 55}
```

### Question answering

Question answering is another token-level task that returns an answer to a question, sometimes with context (open-domain) and other times without context (closed-domain).


key: _node_content
string_value:
{"id_": "5d1ce008-57a2-4902-a770-5915e8eeb5c8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96c126ee-bba7-4229-8e80-bc01423634a0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "662a057f46d7e3476debb5ddf37a77744730eaa129df10537f27353be64c6bbc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c944aede-5256-4448-b9a5-756f4c350880", "node_type": "1", "metadata": {}, "hash": "3e794fd959c9b63e7024ab64f859bce441a06f7db3ea2086fe79372522f0ecde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13297, "end_char_idx": 16982, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This task happens whenever we ask a virtual assistant something like whether a restaurant is open. It can also provide customer or technical support and help search engines retrieve the relevant information you're asking for. 

There are two common types of question answering:

* extractive: given a question and some context, the answer is a span of text from the context the model must extract
* abstractive: given a question and some context, the answer is generated from the context; this approach is handled by the [`Text2TextGenerationPipeline`] instead of the [`QuestionAnsweringPipeline`] shown below


```py
>>> from transformers import pipeline

>>> question_answerer = pipeline(task="question-answering")
>>> preds = question_answerer(
...     question="What is the name of the repository?",
...     context="The name of the repository is huggingface/transformers",
... )
>>> print(
...     f"score: {round(preds['score'], 4)}, start: {preds['start']}, end: {preds['end']}, answer: {preds['answer']}"
... )
score: 0.9327, start: 30, end: 54, answer: huggingface/transformers
```

### Summarization

Summarization creates a shorter version of a text from a longer one while trying to preserve most of the meaning of the original document. Summarization is a sequence-to-sequence task; it outputs a shorter text sequence than the input. There are a lot of long-form documents that can be summarized to help readers quickly understand the main points. Legislative bills, legal and financial documents, patents, and scientific papers are a few examples of documents that could be summarized to save readers time and serve as a reading aid.

Like question answering, there are two types of summarization:

* extractive: identify and extract the most important sentences from the original text
* abstractive: generate the target summary (which may include new words not in the input document) from the original text; the [`SummarizationPipeline`] uses the abstractive approach

```py
>>> from transformers import pipeline

>>> summarizer = pipeline(task="summarization")
>>> summarizer(
...     "In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles."
... )
[{'summary_text': ' The Transformer is the first sequence transduction model based entirely on attention . It replaces the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention . For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers .'}]
```

### Translation

Translation converts a sequence of text in one language to another. It is important in helping people from different backgrounds communicate with each other, help translate content to reach wider audiences, and even be a learning tool to help people learn a new language. Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence. 

In the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages.


key: _node_content
string_value:
{"id_": "c944aede-5256-4448-b9a5-756f4c350880", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5d1ce008-57a2-4902-a770-5915e8eeb5c8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "a00de75ac4f3512ce364808aca0bcdc175b5652b26c21d546348ac9cd30e67fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "86527754-ae04-4ff4-88cc-2617586d7aae", "node_type": "1", "metadata": {}, "hash": "7d418c436298dbea029ff6a7f74cd864b74efeb32e4b425332107fab58fdc26c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14946, "end_char_idx": 18535, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Like question answering, there are two types of summarization:

* extractive: identify and extract the most important sentences from the original text
* abstractive: generate the target summary (which may include new words not in the input document) from the original text; the [`SummarizationPipeline`] uses the abstractive approach

```py
>>> from transformers import pipeline

>>> summarizer = pipeline(task="summarization")
>>> summarizer(
...     "In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles."
... )
[{'summary_text': ' The Transformer is the first sequence transduction model based entirely on attention . It replaces the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention . For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers .'}]
```

### Translation

Translation converts a sequence of text in one language to another. It is important in helping people from different backgrounds communicate with each other, help translate content to reach wider audiences, and even be a learning tool to help people learn a new language. Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence. 

In the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages.

```py
>>> from transformers import pipeline

>>> text = "translate English to French: Hugging Face is a community-based open-source platform for machine learning."
>>> translator = pipeline(task="translation", model="google-t5/t5-small")
>>> translator(text)
[{'translation_text': "Hugging Face est une tribune communautaire de l'apprentissage des machines."}]
```

### Language modeling

Language modeling is a task that predicts a word in a sequence of text. It has become a very popular NLP task because a pretrained language model can be finetuned for many other downstream tasks. Lately, there has been a lot of interest in large language models (LLMs) which demonstrate zero- or few-shot learning. This means the model can solve tasks it wasn't explicitly trained to do! Language models can be used to generate fluent and convincing text, though you need to be careful since the text may not always be accurate.

There are two types of language modeling:

* causal: the model's objective is to predict the next token in a sequence, and future tokens are masked

    ```py
    >>> from transformers import pipeline

    >>> prompt = "Hugging Face is a community-based open-source platform for machine learning."
    >>> generator = pipeline(task="text-generation")
    >>> generator(prompt)  # doctest: +SKIP
    ```

* masked: the model's objective is to predict a masked token in a sequence with full access to the tokens in the sequence
    
    ```py
    >>> text = "Hugging Face is a community-based open-source <mask> for machine learning."


key: _node_content
string_value:
{"id_": "86527754-ae04-4ff4-88cc-2617586d7aae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c944aede-5256-4448-b9a5-756f4c350880", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "35c4896e47ed93ab7b068296a9a40139f63c80acb81a26879fe122c849d93b1c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6565f1a3-a6b3-4638-be6c-826a7ff4044f", "node_type": "1", "metadata": {}, "hash": "0da49853a00474ed338686b9ccff22384b3dcbe7afb95a3a197d607cc3c1f83b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16641, "end_char_idx": 20295, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence. 

In the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages.

```py
>>> from transformers import pipeline

>>> text = "translate English to French: Hugging Face is a community-based open-source platform for machine learning."
>>> translator = pipeline(task="translation", model="google-t5/t5-small")
>>> translator(text)
[{'translation_text': "Hugging Face est une tribune communautaire de l'apprentissage des machines."}]
```

### Language modeling

Language modeling is a task that predicts a word in a sequence of text. It has become a very popular NLP task because a pretrained language model can be finetuned for many other downstream tasks. Lately, there has been a lot of interest in large language models (LLMs) which demonstrate zero- or few-shot learning. This means the model can solve tasks it wasn't explicitly trained to do! Language models can be used to generate fluent and convincing text, though you need to be careful since the text may not always be accurate.

There are two types of language modeling:

* causal: the model's objective is to predict the next token in a sequence, and future tokens are masked

    ```py
    >>> from transformers import pipeline

    >>> prompt = "Hugging Face is a community-based open-source platform for machine learning."
    >>> generator = pipeline(task="text-generation")
    >>> generator(prompt)  # doctest: +SKIP
    ```

* masked: the model's objective is to predict a masked token in a sequence with full access to the tokens in the sequence
    
    ```py
    >>> text = "Hugging Face is a community-based open-source <mask> for machine learning."
    >>> fill_mask = pipeline(task="fill-mask")
    >>> preds = fill_mask(text, top_k=1)
    >>> preds = [
    ...     {
    ...         "score": round(pred["score"], 4),
    ...         "token": pred["token"],
    ...         "token_str": pred["token_str"],
    ...         "sequence": pred["sequence"],
    ...     }
    ...     for pred in preds
    ... ]
    >>> preds
    [{'score': 0.2236,
      'token': 1761,
      'token_str': ' platform',
      'sequence': 'Hugging Face is a community-based open-source platform for machine learning.'}]
    ```

## Multimodal

Multimodal tasks require a model to process multiple data modalities (text, image, audio, video) to solve a particular problem. Image captioning is an example of a multimodal task where the model takes an image as input and outputs a sequence of text describing the image or some properties of the image. 

Although multimodal models work with different data types or modalities, internally, the preprocessing steps help the model convert all the data types into embeddings (vectors or list of numbers that holds meaningful information about the data). For a task like image captioning, the model learns relationships between image embeddings and text embeddings.

### Document question answering

Document question answering is a task that answers natural language questions from a document. Unlike a token-level question answering task which takes text as input, document question answering takes an image of a document as input along with a question about the document and returns an answer. Document question answering can be used to parse structured documents and extract key information from it. In the example below, the total amount and change due can be extracted from a receipt.


key: _node_content
string_value:
{"id_": "6565f1a3-a6b3-4638-be6c-826a7ff4044f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82a98156-a531-459c-8251-4aa525ca11e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "b100e08e42e20bf3cdad6bb92be49a3190daf033cba62810ec3b763166b663f7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "86527754-ae04-4ff4-88cc-2617586d7aae", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/task_summary", "title": "What \ud83e\udd17 Transformers can do", "tokens": 4858, "retrieve_doc": true, "source": "transformers"}, "hash": "54084d551ddb6e20e9b0c3b76e7deb11d59952c86b4cdf7ba8c5c1bf7ba8e4a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18540, "end_char_idx": 21111, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> fill_mask = pipeline(task="fill-mask")
    >>> preds = fill_mask(text, top_k=1)
    >>> preds = [
    ...     {
    ...         "score": round(pred["score"], 4),
    ...         "token": pred["token"],
    ...         "token_str": pred["token_str"],
    ...         "sequence": pred["sequence"],
    ...     }
    ...     for pred in preds
    ... ]
    >>> preds
    [{'score': 0.2236,
      'token': 1761,
      'token_str': ' platform',
      'sequence': 'Hugging Face is a community-based open-source platform for machine learning.'}]
    ```

## Multimodal

Multimodal tasks require a model to process multiple data modalities (text, image, audio, video) to solve a particular problem. Image captioning is an example of a multimodal task where the model takes an image as input and outputs a sequence of text describing the image or some properties of the image. 

Although multimodal models work with different data types or modalities, internally, the preprocessing steps help the model convert all the data types into embeddings (vectors or list of numbers that holds meaningful information about the data). For a task like image captioning, the model learns relationships between image embeddings and text embeddings.

### Document question answering

Document question answering is a task that answers natural language questions from a document. Unlike a token-level question answering task which takes text as input, document question answering takes an image of a document as input along with a question about the document and returns an answer. Document question answering can be used to parse structured documents and extract key information from it. In the example below, the total amount and change due can be extracted from a receipt.

```py
>>> from transformers import pipeline
>>> from PIL import Image
>>> import requests

>>> url = "https://huggingface.co/datasets/hf-internal-testing/example-documents/resolve/main/jpeg_images/2.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> doc_question_answerer = pipeline("document-question-answering", model="magorshunov/layoutlm-invoices")
>>> preds = doc_question_answerer(
...     question="What is the total amount?",
...     image=image,
... )
>>> preds
[{'score': 0.8531, 'answer': '17,000', 'start': 4, 'end': 4}]
```

Hopefully, this page has given you some more background information about all the types of tasks in each modality and the practical importance of each one. In the next [section](tasks_explained), you'll learn **how** 🤗 Transformers work to solve these tasks.


key: _node_content
string_value:
{"id_": "b839ca8e-722c-44e5-ab1a-3085f9db1814", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "7d797771b1f9101ed3be1753409bd31a324784969f01bfec031a47be93a6c7d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bf22f72a-0050-459e-b9c1-eb361f51c590", "node_type": "1", "metadata": {}, "hash": "017ffcb932fff7f132a0bcacc761fc07a8eedf1027a728bf51701e62a23b5d00", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2284, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Hyperparameter Search using Trainer API

🤗 Transformers provides a [`Trainer`] class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The [`Trainer`] provides API for hyperparameter search. This doc shows how to enable it in example. 

## Hyperparameter Search backend

[`Trainer`] supports four hyperparameter search backends currently:
[optuna](https://optuna.org/), [sigopt](https://sigopt.com/), [raytune](https://docs.ray.io/en/latest/tune/index.html) and [wandb](https://wandb.ai/site/sweeps).

you should install them before using them as the hyperparameter search backend
```bash
pip install optuna/sigopt/wandb/ray[tune] 
```

## How to enable Hyperparameter search in example

Define the hyperparameter search space, different backends need different format.

For sigopt, see sigopt [object_parameter](https://docs.sigopt.com/ai-module-api-references/api_reference/objects/object_parameter), it's like following:
```py
>>> def sigopt_hp_space(trial):
...     return [
...         {"bounds": {"min": 1e-6, "max": 1e-4}, "name": "learning_rate", "type": "double"},
...         {
...             "categorical_values": ["16", "32", "64", "128"],
...             "name": "per_device_train_batch_size",
...             "type": "categorical",
...         },
...     ]
```

For optuna, see optuna [object_parameter](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html#sphx-glr-tutorial-10-key-features-002-configurations-py), it's like following:

```py
>>> def optuna_hp_space(trial):
...     return {
...         "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
...         "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [16, 32, 64, 128]),
...     }
```

Optuna provides multi-objective HPO. You can pass `direction` in `hyperparameter_search` and define your own compute_objective to return multiple objective values. The Pareto Front (`List[BestRun]`) will be returned in hyperparameter_search, you should refer to the test case `TrainerHyperParameterMultiObjectOptunaIntegrationTest` in [test_trainer](https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py).


key: _node_content
string_value:
{"id_": "bf22f72a-0050-459e-b9c1-eb361f51c590", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "7d797771b1f9101ed3be1753409bd31a324784969f01bfec031a47be93a6c7d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b839ca8e-722c-44e5-ab1a-3085f9db1814", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "bed946cff9e56eca9bd01f48b6c4e14935230cf29fae29f715f4721e6c0ab747", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b52e1bf6-9fcf-45b3-b774-fd729884615f", "node_type": "1", "metadata": {}, "hash": "8145890f6c49bce81fe9d1fb61705892269fbf1b8174c3093ed66350e7eead1a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 853, "end_char_idx": 4396, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For sigopt, see sigopt [object_parameter](https://docs.sigopt.com/ai-module-api-references/api_reference/objects/object_parameter), it's like following:
```py
>>> def sigopt_hp_space(trial):
...     return [
...         {"bounds": {"min": 1e-6, "max": 1e-4}, "name": "learning_rate", "type": "double"},
...         {
...             "categorical_values": ["16", "32", "64", "128"],
...             "name": "per_device_train_batch_size",
...             "type": "categorical",
...         },
...     ]
```

For optuna, see optuna [object_parameter](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html#sphx-glr-tutorial-10-key-features-002-configurations-py), it's like following:

```py
>>> def optuna_hp_space(trial):
...     return {
...         "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
...         "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [16, 32, 64, 128]),
...     }
```

Optuna provides multi-objective HPO. You can pass `direction` in `hyperparameter_search` and define your own compute_objective to return multiple objective values. The Pareto Front (`List[BestRun]`) will be returned in hyperparameter_search, you should refer to the test case `TrainerHyperParameterMultiObjectOptunaIntegrationTest` in [test_trainer](https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py). It's like following

```py
>>> best_trials = trainer.hyperparameter_search(
...     direction=["minimize", "maximize"],
...     backend="optuna",
...     hp_space=optuna_hp_space,
...     n_trials=20,
...     compute_objective=compute_objective,
... )
```

For raytune, see raytune [object_parameter](https://docs.ray.io/en/latest/tune/api/search_space.html), it's like following:

```py
>>> def ray_hp_space(trial):
...     return {
...         "learning_rate": tune.loguniform(1e-6, 1e-4),
...         "per_device_train_batch_size": tune.choice([16, 32, 64, 128]),
...     }
```

For wandb, see wandb [object_parameter](https://docs.wandb.ai/guides/sweeps/configuration), it's like following:

```py
>>> def wandb_hp_space(trial):
...     return {
...         "method": "random",
...         "metric": {"name": "objective", "goal": "minimize"},
...         "parameters": {
...             "learning_rate": {"distribution": "uniform", "min": 1e-6, "max": 1e-4},
...             "per_device_train_batch_size": {"values": [16, 32, 64, 128]},
...         },
...     }
```

Define a `model_init` function and pass it to the [`Trainer`], as an example:
```py
>>> def model_init(trial):
...     return AutoModelForSequenceClassification.from_pretrained(
...         model_args.model_name_or_path,
...         from_tf=bool(".ckpt" in model_args.model_name_or_path),
...         config=config,
...         cache_dir=model_args.cache_dir,
...         revision=model_args.model_revision,
...         token=True if model_args.use_auth_token else None,
...     )
```

Create a [`Trainer`] with your `model_init` function, training arguments, training and test datasets, and evaluation function:

```py
>>> trainer = Trainer(
...     model=None,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
...     tokenizer=tokenizer,
...     model_init=model_init,
...     data_collator=data_collator,
... )
```

Call hyperparameter search, get the best trial parameters, backend could be `"optuna"`/`"sigopt"`/`"wandb"`/`"ray"`.


key: _node_content
string_value:
{"id_": "b52e1bf6-9fcf-45b3-b774-fd729884615f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1b5376d4-d342-4e26-8840-8ff69c7e9c4f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "7d797771b1f9101ed3be1753409bd31a324784969f01bfec031a47be93a6c7d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bf22f72a-0050-459e-b9c1-eb361f51c590", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/hpo_train", "title": "Hyperparameter Search using Trainer API", "tokens": 1437, "retrieve_doc": true, "source": "transformers"}, "hash": "fd4ac6bd272e6f7923fb7bc19e055aa2ed159d27d69db3cff828897d8f462865", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4397, "end_char_idx": 5125, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
direction can be`"minimize"` or `"maximize"`, which indicates whether to optimize greater or lower objective.

You could define your own compute_objective function, if not defined, the default compute_objective will be called, and the sum of eval metric like f1 is returned as objective value.

```py
>>> best_trial = trainer.hyperparameter_search(
...     direction="maximize",
...     backend="optuna",
...     hp_space=optuna_hp_space,
...     n_trials=20,
...     compute_objective=compute_objective,
... )
```

## Hyperparameter search For DDP finetune
Currently, Hyperparameter search for DDP is enabled for optuna and sigopt. Only the rank-zero process will generate the search trial and pass the argument to other ranks.


key: _node_content
string_value:
{"id_": "2e22c5e9-1bf6-4a1c-9509-01a2ebfa1202", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "79df0831-2d78-4f1a-99f3-69fac4aa6ba4", "node_type": "1", "metadata": {}, "hash": "005e41af40faf253ff851d022474fb9fc54a4ed3782803c354845a565c76f040", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Efficient Training on Multiple GPUs

If training a model on a single GPU is too slow or if the model's weights do not fit in a single GPU's memory, transitioning 
to a multi-GPU setup may be a viable option. Prior to making this transition, thoroughly explore all the strategies covered 
in the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one) as they are universally applicable 
to model training on any number of GPUs. Once you have employed those strategies and found them insufficient for your 
case on a single GPU, consider moving to multiple GPUs.

Transitioning from a single GPU to multiple GPUs requires the introduction of some form of parallelism, as the workload 
must be distributed across the resources. Multiple techniques can be employed to achieve parallelism, such as data 
parallelism, tensor parallelism, and pipeline parallelism. It's important to note that there isn't a one-size-fits-all 
solution, and the optimal settings depend on the specific hardware configuration you are using. 

This guide offers an in-depth overview of individual types of parallelism, as well as guidance on ways to combine   
techniques and choosing an appropriate approach. For step-by-step tutorials on distributed training, please refer to
the [🤗 Accelerate documentation](https://huggingface.co/docs/accelerate/index). 

<Tip>

While the main concepts discussed in this guide are likely applicable across frameworks, here we focus on 
PyTorch-based implementations.

</Tip>

Before diving deeper into the specifics of each technique, let's go over the rough decision process when training 
large models on a large infrastructure.

## Scalability strategy

Begin by estimating how much vRAM is required to train your model. For models hosted on the 🤗 Hub, use our 
[Model Memory Calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage), which gives you 
accurate calculations within a few percent margin.  

**Parallelization strategy for a single Node / multi-GPU setup**

When training a model on a single node with multiple GPUs, your choice of parallelization strategy can significantly 
impact performance. Here's a breakdown of your options:

**Case 1: Your model fits onto a single GPU**

If your model can comfortably fit onto a single GPU, you have two primary options:

1. DDP - Distributed DataParallel
2. [Zero Redundancy Optimizer (ZeRO)](https://arxiv.org/abs/1910.02054) - depending on the situation and configuration used, this method may or may not be faster, however, it's worth experimenting with it.

**Case 2: Your model doesn't fit onto a single GPU:**

If your model is too large for a single GPU, you have several alternatives to consider:

1. PipelineParallel (PP)
2. [ZeRO](https://arxiv.org/abs/1910.02054)
3. [TensorParallel](#tensor-parallelism) (TP)

With very fast inter-node connectivity (e.g., NVLINK or NVSwitch) all three strategies (PP, ZeRO, TP) should result in 
similar performance. However, without these, PP will be faster than TP or ZeRO. The degree of TP may also 
make a difference. It's best to experiment with your specific setup to determine the most suitable strategy.

TP is almost always used within a single node. That is TP size <= GPUs per node.

**Case 3: Largest layer of your model does not fit onto a single GPU**

1.


key: _node_content
string_value:
{"id_": "79df0831-2d78-4f1a-99f3-69fac4aa6ba4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e22c5e9-1bf6-4a1c-9509-01a2ebfa1202", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "edda93c2c16079dafb7ff6b27bd0bcfd8e4265d353887ce53172e1b2412a8cdd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4222c35-a1cd-4027-b488-bf4ef5a4280e", "node_type": "1", "metadata": {}, "hash": "b48e0432309b056879d994ae6b0d5a17be52371a5d7e868384d54788ca932a74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1761, "end_char_idx": 4875, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For models hosted on the 🤗 Hub, use our 
[Model Memory Calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage), which gives you 
accurate calculations within a few percent margin.  

**Parallelization strategy for a single Node / multi-GPU setup**

When training a model on a single node with multiple GPUs, your choice of parallelization strategy can significantly 
impact performance. Here's a breakdown of your options:

**Case 1: Your model fits onto a single GPU**

If your model can comfortably fit onto a single GPU, you have two primary options:

1. DDP - Distributed DataParallel
2. [Zero Redundancy Optimizer (ZeRO)](https://arxiv.org/abs/1910.02054) - depending on the situation and configuration used, this method may or may not be faster, however, it's worth experimenting with it.

**Case 2: Your model doesn't fit onto a single GPU:**

If your model is too large for a single GPU, you have several alternatives to consider:

1. PipelineParallel (PP)
2. [ZeRO](https://arxiv.org/abs/1910.02054)
3. [TensorParallel](#tensor-parallelism) (TP)

With very fast inter-node connectivity (e.g., NVLINK or NVSwitch) all three strategies (PP, ZeRO, TP) should result in 
similar performance. However, without these, PP will be faster than TP or ZeRO. The degree of TP may also 
make a difference. It's best to experiment with your specific setup to determine the most suitable strategy.

TP is almost always used within a single node. That is TP size <= GPUs per node.

**Case 3: Largest layer of your model does not fit onto a single GPU**

1. If you are not using ZeRO, you have to use TensorParallel (TP), because PipelineParallel (PP) alone won't be sufficient to accommodate the large layer.
2. If you are using ZeRO, additionally adopt techniques from the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one).

**Parallelization strategy for a multi-Node / multi-GPU setup**

* When you have fast inter-node connectivity (e.g., NVLINK or NVSwitch) consider using one of these options:

    1. ZeRO - as it requires close to no modifications to the model
    2. A combination of PipelineParallel(PP) with TensorParallel(TP) and DataParallel(DP) - this approach will result in fewer communications, but requires significant changes to the model

* When you have slow inter-node connectivity and still low on GPU memory:

    1. Employ a combination of DataParallel(DP) with PipelineParallel(PP), TensorParallel(TP), and ZeRO.

In the following sections of this guide we dig deeper into how these different parallelism methods work.

## Data Parallelism

Even with only 2 GPUs, you can readily leverage the accelerated training capabilities offered by PyTorch's built-in features, 
such as `DataParallel` (DP) and `DistributedDataParallel` (DDP). Note that 
[PyTorch documentation](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html) recommends to prefer 
`DistributedDataParallel` (DDP) over `DataParallel` (DP) for multi-GPU training as it works for all models.
Let's take a look at how these two methods work and what makes them different.


key: _node_content
string_value:
{"id_": "e4222c35-a1cd-4027-b488-bf4ef5a4280e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "79df0831-2d78-4f1a-99f3-69fac4aa6ba4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "6b696a993028743669c1db7d105a69818b9a60ac643ccf3bb2bb192143b7a516", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "309298b5-445d-46bd-8b79-1604f0c25ead", "node_type": "1", "metadata": {}, "hash": "bf5757eb6a7e4fcfc7abc76d7017f285b7f99b1c77e398a6ae64f7d2a18eabe1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3174, "end_char_idx": 6470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
TP is almost always used within a single node. That is TP size <= GPUs per node.

**Case 3: Largest layer of your model does not fit onto a single GPU**

1. If you are not using ZeRO, you have to use TensorParallel (TP), because PipelineParallel (PP) alone won't be sufficient to accommodate the large layer.
2. If you are using ZeRO, additionally adopt techniques from the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one).

**Parallelization strategy for a multi-Node / multi-GPU setup**

* When you have fast inter-node connectivity (e.g., NVLINK or NVSwitch) consider using one of these options:

    1. ZeRO - as it requires close to no modifications to the model
    2. A combination of PipelineParallel(PP) with TensorParallel(TP) and DataParallel(DP) - this approach will result in fewer communications, but requires significant changes to the model

* When you have slow inter-node connectivity and still low on GPU memory:

    1. Employ a combination of DataParallel(DP) with PipelineParallel(PP), TensorParallel(TP), and ZeRO.

In the following sections of this guide we dig deeper into how these different parallelism methods work.

## Data Parallelism

Even with only 2 GPUs, you can readily leverage the accelerated training capabilities offered by PyTorch's built-in features, 
such as `DataParallel` (DP) and `DistributedDataParallel` (DDP). Note that 
[PyTorch documentation](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html) recommends to prefer 
`DistributedDataParallel` (DDP) over `DataParallel` (DP) for multi-GPU training as it works for all models.
Let's take a look at how these two methods work and what makes them different.

### DataParallel vs DistributedDataParallel

To understand the key differences in inter-GPU communication overhead between the two methods, let's review the processes per batch:

[DDP](https://pytorch.org/docs/master/notes/ddp.html):

- At the start time the main process replicates the model once from GPU 0 to the rest of GPUs
- Then for each batch:
   1. Each GPU directly consumes its mini-batch of data.
   2. During `backward`, once the local gradients are ready, they are averaged across all processes.

[DP](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html):

For each batch:
   1. GPU 0 reads the batch of data and then sends a mini-batch to each GPU.
   2. The up-to-date model is replicated from GPU 0 to each GPU. 
   3. `forward` is executed, and output from each GPU is sent to GPU 0 to compute the loss.
   4. The loss is distributed from GPU 0 to all GPUs, and `backward` is run. 
   5. Gradients from each GPU are sent to GPU 0 and averaged. 

Key differences include:
1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.
DDP copies data using [torch.distributed](https://pytorch.org/docs/master/distributed.html), while DP copies data within 
the process via Python threads (which introduces limitations associated with GIL). As a result, **`DistributedDataParallel` (DDP) is generally faster than `DataParallel` (DP)** unless you have slow GPU card inter-connectivity.
2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization.


key: _node_content
string_value:
{"id_": "309298b5-445d-46bd-8b79-1604f0c25ead", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4222c35-a1cd-4027-b488-bf4ef5a4280e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "b1a6356698a0e08d26b9c013e52556c6de384f6863d8d15fc894d10c3bd9240d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a362dc7-85c9-435b-9d60-4e3656f0396d", "node_type": "1", "metadata": {}, "hash": "bebf73d163113ba00616cde8f96a8aaaa5c45c1401609b5cd9e2b64cd8d04f06", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4877, "end_char_idx": 7273, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### DataParallel vs DistributedDataParallel

To understand the key differences in inter-GPU communication overhead between the two methods, let's review the processes per batch:

[DDP](https://pytorch.org/docs/master/notes/ddp.html):

- At the start time the main process replicates the model once from GPU 0 to the rest of GPUs
- Then for each batch:
   1. Each GPU directly consumes its mini-batch of data.
   2. During `backward`, once the local gradients are ready, they are averaged across all processes.

[DP](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html):

For each batch:
   1. GPU 0 reads the batch of data and then sends a mini-batch to each GPU.
   2. The up-to-date model is replicated from GPU 0 to each GPU. 
   3. `forward` is executed, and output from each GPU is sent to GPU 0 to compute the loss.
   4. The loss is distributed from GPU 0 to all GPUs, and `backward` is run. 
   5. Gradients from each GPU are sent to GPU 0 and averaged. 

Key differences include:
1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.
DDP copies data using [torch.distributed](https://pytorch.org/docs/master/distributed.html), while DP copies data within 
the process via Python threads (which introduces limitations associated with GIL). As a result, **`DistributedDataParallel` (DDP) is generally faster than `DataParallel` (DP)** unless you have slow GPU card inter-connectivity.
2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization. 
3. DDP supports distributed training across multiple machines, whereas DP does not.

This is not an exhaustive list of differences between DP and DDP, however, other nuances are out of scope of this guide.
You can get a deeper understanding of these methods by reading this [article](https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/).

Let's illustrate the differences between DP and DDP with an experiment. We'll benchmark the differences between DP and 
DDP with an added context of NVLink presence:  

* Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (`NV2` in `nvidia-smi topo -m`).
* Software: `pytorch-1.8-to-be` + `cuda-11.0` / `transformers==4.3.0.dev0`.

To disable the NVLink feature on one of the benchmarks, we use `NCCL_P2P_DISABLE=1`.


key: _node_content
string_value:
{"id_": "4a362dc7-85c9-435b-9d60-4e3656f0396d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "309298b5-445d-46bd-8b79-1604f0c25ead", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "47aca6886d2272ced149346b81b34c2dc8d661a9ae3ff4a99f6be451a032eab9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e86d998-02a5-4c8b-bd31-46a3466ec2b4", "node_type": "1", "metadata": {}, "hash": "85b9fa7888aa9d5271446a1e78e454e8f04022fe55412e29990e292f952b5cdb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5719, "end_char_idx": 8920, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4. The loss is distributed from GPU 0 to all GPUs, and `backward` is run. 
   5. Gradients from each GPU are sent to GPU 0 and averaged. 

Key differences include:
1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.
DDP copies data using [torch.distributed](https://pytorch.org/docs/master/distributed.html), while DP copies data within 
the process via Python threads (which introduces limitations associated with GIL). As a result, **`DistributedDataParallel` (DDP) is generally faster than `DataParallel` (DP)** unless you have slow GPU card inter-connectivity.
2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization. 
3. DDP supports distributed training across multiple machines, whereas DP does not.

This is not an exhaustive list of differences between DP and DDP, however, other nuances are out of scope of this guide.
You can get a deeper understanding of these methods by reading this [article](https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/).

Let's illustrate the differences between DP and DDP with an experiment. We'll benchmark the differences between DP and 
DDP with an added context of NVLink presence:  

* Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (`NV2` in `nvidia-smi topo -m`).
* Software: `pytorch-1.8-to-be` + `cuda-11.0` / `transformers==4.3.0.dev0`.

To disable the NVLink feature on one of the benchmarks, we use `NCCL_P2P_DISABLE=1`. 

Here is the benchmarking code and outputs:

**DP**

```bash
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
python examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 110.5948, 'train_samples_per_second': 1.808, 'epoch': 0.69}
```

**DDP w/ NVlink**

```bash
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
torchrun --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 101.9003, 'train_samples_per_second': 1.963, 'epoch': 0.69}
```

**DDP w/o NVlink**

```bash
rm -r /tmp/test-clm; NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1 \
torchrun --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 131.4367, 'train_samples_per_second': 1.522, 'epoch': 0.69}
```

Here are the same benchmarking results gathered in a table for convenience:

| Type   | NVlink | Time |
| :----- | -----  | ---: |
| 2:DP   | Y      | 110s |
| 2:DDP  | Y      | 101s |
| 2:DDP  | N      | 131s |

As you can see, in this case DP is ~10% slower than DDP with NVlink, but ~15% faster than DDP without NVlink.


key: _node_content
string_value:
{"id_": "5e86d998-02a5-4c8b-bd31-46a3466ec2b4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a362dc7-85c9-435b-9d60-4e3656f0396d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "5670acada383953423275afbd73606c9ab38bdce9ce386abe45c3f5b64939b8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7cda6709-6a25-4eb9-9871-05c68be2550f", "node_type": "1", "metadata": {}, "hash": "378466ac7060365bbec651c1916cacb9d2cf1733d494ac3482c81a50a344fcaf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8921, "end_char_idx": 11691, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The real difference will depend on how much data each GPU needs to sync with the others - the more there is to sync, 
the more a slow link will impede the overall runtime.

## ZeRO Data Parallelism

ZeRO-powered data parallelism (ZeRO-DP) is illustrated in the following diagram from this [blog post](https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/).

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-zero.png" alt="DeepSpeed-Image-1"/>
 </div>

While it may appear complex, it is a very similar concept to `DataParallel` (DP). The difference is that instead of 
replicating the full model parameters, gradients and optimizer states, each GPU stores only a slice of it. Then, at 
run-time when the full layer parameters are needed just for the given layer, all GPUs synchronize to give each other 
parts that they miss.

To illustrate this idea, consider a simple model with 3 layers (La, Lb, and Lc), where each layer has 3 parameters. 
Layer La, for example, has weights a0, a1 and a2:

```
La | Lb | Lc
---|----|---
a0 | b0 | c0
a1 | b1 | c1
a2 | b2 | c2
```

If we have 3 GPUs, ZeRO-DP splits the model onto 3 GPUs like so:

```
GPU0:
La | Lb | Lc
---|----|---
a0 | b0 | c0

GPU1:
La | Lb | Lc
---|----|---
a1 | b1 | c1

GPU2:
La | Lb | Lc
---|----|---
a2 | b2 | c2
```

In a way, this is the same horizontal slicing as tensor parallelism, as opposed to Vertical 
slicing, where one puts whole layer-groups on different GPUs. Now let's see how this works: 

Each of these GPUs will get the usual mini-batch as it works in DP:

```
x0 => GPU0
x1 => GPU1
x2 => GPU2
```

The inputs are passed without modifications as if they would be processed by the original model.

First, the inputs get to the layer `La`. What happens at this point?

On GPU0: the x0 mini-batch requires the a0, a1, a2 parameters to do its forward path through the layer, but the GPU0 has only a0. 
It will get a1 from GPU1 and a2 from GPU2, bringing all the pieces of the model together.

In parallel, GPU1 gets another mini-batch - x1. GPU1 has the a1 parameter, but needs a0 and a2, so it gets those from GPU0 and GPU2.
Same happens to GPU2 that gets the mini-batch x2. It gets a0 and a1 from GPU0 and GPU1.

This way each of the 3 GPUs gets the full tensors reconstructed and makes a forward pass with its own mini-batch.
As soon as the calculation is done, the data that is no longer needed gets dropped - it's only used during the calculation. 
The reconstruction is done efficiently via a pre-fetch.

Then the whole process is repeated for layer Lb, then Lc forward-wise, and then backward Lc -> Lb -> La.


key: _node_content
string_value:
{"id_": "7cda6709-6a25-4eb9-9871-05c68be2550f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e86d998-02a5-4c8b-bd31-46a3466ec2b4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "ba2ec91a20f64213ab43a86eb2bef4b4b7405e40a77cd46c93edc379845389b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c4a4ffe-2df7-45dc-b7fa-9b51540219ef", "node_type": "1", "metadata": {}, "hash": "39be32304d025f8427f9a7fa59cceb63b5213d09d8f261d9ebdd90c787ccd727", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10536, "end_char_idx": 13444, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now let's see how this works: 

Each of these GPUs will get the usual mini-batch as it works in DP:

```
x0 => GPU0
x1 => GPU1
x2 => GPU2
```

The inputs are passed without modifications as if they would be processed by the original model.

First, the inputs get to the layer `La`. What happens at this point?

On GPU0: the x0 mini-batch requires the a0, a1, a2 parameters to do its forward path through the layer, but the GPU0 has only a0. 
It will get a1 from GPU1 and a2 from GPU2, bringing all the pieces of the model together.

In parallel, GPU1 gets another mini-batch - x1. GPU1 has the a1 parameter, but needs a0 and a2, so it gets those from GPU0 and GPU2.
Same happens to GPU2 that gets the mini-batch x2. It gets a0 and a1 from GPU0 and GPU1.

This way each of the 3 GPUs gets the full tensors reconstructed and makes a forward pass with its own mini-batch.
As soon as the calculation is done, the data that is no longer needed gets dropped - it's only used during the calculation. 
The reconstruction is done efficiently via a pre-fetch.

Then the whole process is repeated for layer Lb, then Lc forward-wise, and then backward Lc -> Lb -> La.

<Tip>

This mechanism is similar to an efficient group backpacking strategy: person A carries the tent, person B carries the stove,
and person C carries the axe. Each night they all share what they have with others and get from others what they don't have, 
and in the morning they pack up their allocated type of gear and continue on their way. This is what ZeRO DP/Sharded DDP is.
Compare this strategy to the simple one where each person has to carry their own tent, stove and axe (similar to 
DataParallel (DP and DDP) in PyTorch), which would be far more inefficient. 

</Tip>

While reading the literature on this topic you may encounter the following synonyms: Sharded, Partitioned.
If you pay close attention the way ZeRO partitions the model's weights - it looks very similar to tensor parallelism 
which will be discussed later. This is because it partitions/shards each layer's weights, unlike vertical model parallelism 
which is discussed next.

Implementations:

- [DeepSpeed](https://www.deepspeed.ai/tutorials/zero/) ZeRO-DP stages 1+2+3
- [`Accelerate` integration](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed) 
- [`transformers` integration](main_classes/trainer#trainer-integrations)

## From Naive Model Parallelism to Pipeline Parallelism

To explain Pipeline parallelism, we'll first look into Naive Model Parallelism (MP), also known as Vertical MP. This approach
involves distributing groups of model layers across multiple GPUs by assigning specific layers to specific GPUs with `.to()`. 
As data flows through these layers, it is moved to the same GPU as the layer, while the other layers remain untouched.

We refer to this Model parallelism as "Vertical" because of how models are typically visualized.


key: _node_content
string_value:
{"id_": "8c4a4ffe-2df7-45dc-b7fa-9b51540219ef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7cda6709-6a25-4eb9-9871-05c68be2550f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "f94c80639eb959d39985d655b1eaddf316029daf6266529a439f0e02ad64eb28", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1d932fcf-f854-410d-9398-43ae4b44a197", "node_type": "1", "metadata": {}, "hash": "88288d70eac682fa5f9e040976555b7a9d20dc293bf1a83ffae53da30b85456a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11693, "end_char_idx": 14883, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

This mechanism is similar to an efficient group backpacking strategy: person A carries the tent, person B carries the stove,
and person C carries the axe. Each night they all share what they have with others and get from others what they don't have, 
and in the morning they pack up their allocated type of gear and continue on their way. This is what ZeRO DP/Sharded DDP is.
Compare this strategy to the simple one where each person has to carry their own tent, stove and axe (similar to 
DataParallel (DP and DDP) in PyTorch), which would be far more inefficient. 

</Tip>

While reading the literature on this topic you may encounter the following synonyms: Sharded, Partitioned.
If you pay close attention the way ZeRO partitions the model's weights - it looks very similar to tensor parallelism 
which will be discussed later. This is because it partitions/shards each layer's weights, unlike vertical model parallelism 
which is discussed next.

Implementations:

- [DeepSpeed](https://www.deepspeed.ai/tutorials/zero/) ZeRO-DP stages 1+2+3
- [`Accelerate` integration](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed) 
- [`transformers` integration](main_classes/trainer#trainer-integrations)

## From Naive Model Parallelism to Pipeline Parallelism

To explain Pipeline parallelism, we'll first look into Naive Model Parallelism (MP), also known as Vertical MP. This approach
involves distributing groups of model layers across multiple GPUs by assigning specific layers to specific GPUs with `.to()`. 
As data flows through these layers, it is moved to the same GPU as the layer, while the other layers remain untouched.

We refer to this Model parallelism as "Vertical" because of how models are typically visualized. For example, the 
following diagram shows an 8-layer model split vertically into two slices, placing layers 0-3 onto 
GPU0 and 4-7 to GPU1:

```
================
| Layer |      |
|   0   |      |
|   1   | GPU0 |
|   2   |      |
|   3   |      |
================
| Layer |      |
|   4   |      |
|   5   | GPU1 |
|   6   |      |
|   7   |      |
================
```

In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data 
from layer 3 to 4 requires moving it from GPU0 to GPU1, introducing a communication overhead. If the participating 
GPUs are on the same compute node (e.g. same physical machine) this copying is fast, but if the GPUs are distributed 
across different compute nodes (e.g. multiple machines), the communication overhead could be substantially greater.

Following that, layers 4 to 7 work as they would in the original model. Upon completion of the 7th layer, there is often 
a need to send the data back to layer 0 where the labels are (or alternatively send the labels to the last layer). Now the loss can be 
computed and the optimizer can do its work.

Naive Model Parallelism comes several shortcomings:
- **All but one GPU are idle at any given moment**: if 4 GPUs are used, it's nearly identical to quadrupling the amount of memory of a single GPU, and ignoring the rest of the hardware. 
- **Overhead in data transfer between devices**:  E.g.


key: _node_content
string_value:
{"id_": "1d932fcf-f854-410d-9398-43ae4b44a197", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c4a4ffe-2df7-45dc-b7fa-9b51540219ef", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "23084d12f13ed7910f74aca49a5628c97634c087c9fbab4687d037a02bd4e8cf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4bad0ca4-3813-4657-95f0-d16fd5fe3b6b", "node_type": "1", "metadata": {}, "hash": "f21456a3a798fa9cd866e2e8e3633b38883697b0ac667f86952cf9ab85ef6a51", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13348, "end_char_idx": 16360, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We refer to this Model parallelism as "Vertical" because of how models are typically visualized. For example, the 
following diagram shows an 8-layer model split vertically into two slices, placing layers 0-3 onto 
GPU0 and 4-7 to GPU1:

```
================
| Layer |      |
|   0   |      |
|   1   | GPU0 |
|   2   |      |
|   3   |      |
================
| Layer |      |
|   4   |      |
|   5   | GPU1 |
|   6   |      |
|   7   |      |
================
```

In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data 
from layer 3 to 4 requires moving it from GPU0 to GPU1, introducing a communication overhead. If the participating 
GPUs are on the same compute node (e.g. same physical machine) this copying is fast, but if the GPUs are distributed 
across different compute nodes (e.g. multiple machines), the communication overhead could be substantially greater.

Following that, layers 4 to 7 work as they would in the original model. Upon completion of the 7th layer, there is often 
a need to send the data back to layer 0 where the labels are (or alternatively send the labels to the last layer). Now the loss can be 
computed and the optimizer can do its work.

Naive Model Parallelism comes several shortcomings:
- **All but one GPU are idle at any given moment**: if 4 GPUs are used, it's nearly identical to quadrupling the amount of memory of a single GPU, and ignoring the rest of the hardware. 
- **Overhead in data transfer between devices**:  E.g. 4x 6GB cards will be able to accommodate the same size as 1x 24GB card using naive MP, but a single 24GB card will complete the training faster, because it doesn't have the data copying overhead. But, say, if you have 40GB cards and need to fit a 45GB model you can with 4x 40GB cards (but barely because of the gradient and optimizer states)
- **Copying shared embeddings**: Shared embeddings may need to get copied back and forth between GPUs.

Now that you are familiar with how the naive approach to model parallelism works and its shortcomings, let's look at Pipeline Parallelism (PP).
PP is almost identical to a naive MP, but it solves the GPU idling problem by chunking the incoming batch into micro-batches 
and artificially creating a pipeline, which allows different GPUs to concurrently participate in the computation process.

The following illustration from the [GPipe paper](https://ai.googleblog.com/2019/03/introducing-gpipe-open-source-library.html) 
shows the naive MP on the top, and PP on the bottom:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-gpipe-bubble.png" alt="MP vs PP"/>
</div>

At the bottom of the diagram, you can observe that the Pipeline Parallelism (PP) approach minimizes the number of idle 
GPU zones, referred to as 'bubbles'. Both parts of the diagram show a parallelism level of degree 4, meaning that 4 GPUs 
are involved in the pipeline.


key: _node_content
string_value:
{"id_": "4bad0ca4-3813-4657-95f0-d16fd5fe3b6b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1d932fcf-f854-410d-9398-43ae4b44a197", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "7620cd2a4f2305f994b83300dc7d579bcd44dfb60c72420bbb578e7e55c6336a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4eecdbe4-c5db-4e97-aed2-f92a1be5af4e", "node_type": "1", "metadata": {}, "hash": "6f6b40adaaee90b28b6ff7e08d62b460cbe23ae996936bef9639f66f0631b33b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14829, "end_char_idx": 17787, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- **Overhead in data transfer between devices**:  E.g. 4x 6GB cards will be able to accommodate the same size as 1x 24GB card using naive MP, but a single 24GB card will complete the training faster, because it doesn't have the data copying overhead. But, say, if you have 40GB cards and need to fit a 45GB model you can with 4x 40GB cards (but barely because of the gradient and optimizer states)
- **Copying shared embeddings**: Shared embeddings may need to get copied back and forth between GPUs.

Now that you are familiar with how the naive approach to model parallelism works and its shortcomings, let's look at Pipeline Parallelism (PP).
PP is almost identical to a naive MP, but it solves the GPU idling problem by chunking the incoming batch into micro-batches 
and artificially creating a pipeline, which allows different GPUs to concurrently participate in the computation process.

The following illustration from the [GPipe paper](https://ai.googleblog.com/2019/03/introducing-gpipe-open-source-library.html) 
shows the naive MP on the top, and PP on the bottom:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-gpipe-bubble.png" alt="MP vs PP"/>
</div>

At the bottom of the diagram, you can observe that the Pipeline Parallelism (PP) approach minimizes the number of idle 
GPU zones, referred to as 'bubbles'. Both parts of the diagram show a parallelism level of degree 4, meaning that 4 GPUs 
are involved in the pipeline. You can see that there's a forward path of 4 pipe stages (F0, F1, F2 and F3) followed by 
a backward path in reverse order (B3, B2, B1, and B0).

PP introduces a new hyperparameter to tune - `chunks`, which determines how many data chunks are sent in a sequence 
through the same pipe stage. For example, in the bottom diagram you can see `chunks=4`. GPU0 performs the same 
forward path on chunk 0, 1, 2 and 3 (F0,0, F0,1, F0,2, F0,3) and then it waits for other GPUs to do complete their work. 
Only when the other GPUs begin to complete their work, GPU0 starts to work again doing the backward path for chunks 
3, 2, 1 and 0 (B0,3, B0,2, B0,1, B0,0).

Note that this is the same concept as gradient accumulation steps. PyTorch uses `chunks`, while DeepSpeed refers 
to the same hyperparameter as gradient accumulation steps.

Because of the chunks, PP introduces the notion of micro-batches (MBS). DP splits the global data batch size into 
mini-batches, so if you have a DP degree of 4, a global batch size of 1024 gets split up into 4 mini-batches of 
256 each (1024/4). And if the number of `chunks` (or GAS) is 32 we end up with a micro-batch size of 8 (256/32). Each 
Pipeline stage works with a single micro-batch at a time. To calculate the global batch size of the DP + PP setup, 
use the formula: `mbs * chunks * dp_degree` (`8 * 32 * 4 = 1024`).
With `chunks=1` you end up with the naive MP, which is inefficient.


key: _node_content
string_value:
{"id_": "4eecdbe4-c5db-4e97-aed2-f92a1be5af4e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4bad0ca4-3813-4657-95f0-d16fd5fe3b6b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "3e906ec450eeb649c122789b632a3e5695c37c4b96cac42b54dd3c4635e0a53f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9a025a62-d1f5-4717-a9d3-b4c8879f7763", "node_type": "1", "metadata": {}, "hash": "4bae61f592bceaec4da24c3c774a5e8315cba854e4f1fb2708c619d07aef88d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16507, "end_char_idx": 19513, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
PP introduces a new hyperparameter to tune - `chunks`, which determines how many data chunks are sent in a sequence 
through the same pipe stage. For example, in the bottom diagram you can see `chunks=4`. GPU0 performs the same 
forward path on chunk 0, 1, 2 and 3 (F0,0, F0,1, F0,2, F0,3) and then it waits for other GPUs to do complete their work. 
Only when the other GPUs begin to complete their work, GPU0 starts to work again doing the backward path for chunks 
3, 2, 1 and 0 (B0,3, B0,2, B0,1, B0,0).

Note that this is the same concept as gradient accumulation steps. PyTorch uses `chunks`, while DeepSpeed refers 
to the same hyperparameter as gradient accumulation steps.

Because of the chunks, PP introduces the notion of micro-batches (MBS). DP splits the global data batch size into 
mini-batches, so if you have a DP degree of 4, a global batch size of 1024 gets split up into 4 mini-batches of 
256 each (1024/4). And if the number of `chunks` (or GAS) is 32 we end up with a micro-batch size of 8 (256/32). Each 
Pipeline stage works with a single micro-batch at a time. To calculate the global batch size of the DP + PP setup, 
use the formula: `mbs * chunks * dp_degree` (`8 * 32 * 4 = 1024`).
With `chunks=1` you end up with the naive MP, which is inefficient. With a large `chunks` value you end up with 
tiny micro-batch sizes which is also inefficient. For this reason, we encourage to experiment with the `chunks` value to 
find the one that leads to the most efficient GPUs utilization.

You may notice a bubble of "dead" time on the diagram that can't be parallelized because the last `forward` stage 
has to wait for `backward` to complete the pipeline. The purpose of finding the best value for `chunks` is to enable a high 
concurrent GPU utilization across all participating GPUs which translates to minimizing the size of the bubble.

Pipeline API solutions have been implemented in:
- PyTorch
- DeepSpeed
- Megatron-LM

These come with some shortcomings:
- They have to modify the model quite heavily, because Pipeline requires one to rewrite the normal flow of modules into a `nn.Sequential` sequence of the same, which may require changes to the design of the model.
- Currently the Pipeline API is very restricted. If you had a bunch of Python variables being passed in the very first stage of the Pipeline, you will have to find a way around it. Currently, the pipeline interface requires either a single Tensor or a tuple of Tensors as the only input and output. These tensors must have a batch size as the very first dimension, since pipeline is going to chunk the mini batch into micro-batches. Possible improvements are being discussed here https://github.com/pytorch/pytorch/pull/50693
- Conditional control flow at the level of pipe stages is not possible - e.g., Encoder-Decoder models like T5 require special workarounds to handle a conditional encoder stage.
- They have to arrange each layer so that the output of one layer becomes an input to the other layer.


key: _node_content
string_value:
{"id_": "9a025a62-d1f5-4717-a9d3-b4c8879f7763", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4eecdbe4-c5db-4e97-aed2-f92a1be5af4e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "decb13d3622f891d4f86e59839e97be00432f168b82fdbbcd5e56ee7e98b7cdf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b740f795-1b07-405b-9f74-526d876991cd", "node_type": "1", "metadata": {}, "hash": "c195026b5a467d8dff6cde01624b0c46f0b525bc16611e79a1a01f53ce005fc3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17720, "end_char_idx": 20849, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With `chunks=1` you end up with the naive MP, which is inefficient. With a large `chunks` value you end up with 
tiny micro-batch sizes which is also inefficient. For this reason, we encourage to experiment with the `chunks` value to 
find the one that leads to the most efficient GPUs utilization.

You may notice a bubble of "dead" time on the diagram that can't be parallelized because the last `forward` stage 
has to wait for `backward` to complete the pipeline. The purpose of finding the best value for `chunks` is to enable a high 
concurrent GPU utilization across all participating GPUs which translates to minimizing the size of the bubble.

Pipeline API solutions have been implemented in:
- PyTorch
- DeepSpeed
- Megatron-LM

These come with some shortcomings:
- They have to modify the model quite heavily, because Pipeline requires one to rewrite the normal flow of modules into a `nn.Sequential` sequence of the same, which may require changes to the design of the model.
- Currently the Pipeline API is very restricted. If you had a bunch of Python variables being passed in the very first stage of the Pipeline, you will have to find a way around it. Currently, the pipeline interface requires either a single Tensor or a tuple of Tensors as the only input and output. These tensors must have a batch size as the very first dimension, since pipeline is going to chunk the mini batch into micro-batches. Possible improvements are being discussed here https://github.com/pytorch/pytorch/pull/50693
- Conditional control flow at the level of pipe stages is not possible - e.g., Encoder-Decoder models like T5 require special workarounds to handle a conditional encoder stage.
- They have to arrange each layer so that the output of one layer becomes an input to the other layer.

More recent solutions include:
- Varuna
- Sagemaker

We have not experimented with Varuna and SageMaker but their papers report that they have overcome the list of problems 
mentioned above and that they require smaller changes to the user's model.

Implementations:
- [PyTorch](https://pytorch.org/docs/stable/pipeline.html) (initial support in pytorch-1.8, and progressively getting improved in 1.9 and more so in 1.10). Some [examples](https://github.com/pytorch/pytorch/blob/master/benchmarks/distributed/pipeline/pipe.py)
- [DeepSpeed](https://www.deepspeed.ai/tutorials/pipeline/)
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation - no API.
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.
- [OSLO](https://github.com/tunib-ai/oslo) - this is implemented based on the Hugging Face Transformers.

🤗 Transformers status: as of this writing none of the models supports full-PP. GPT2 and T5 models have naive MP support. 
The main obstacle is being unable to convert the models to `nn.Sequential` and have all the inputs to be Tensors. This 
is because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.


key: _node_content
string_value:
{"id_": "b740f795-1b07-405b-9f74-526d876991cd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9a025a62-d1f5-4717-a9d3-b4c8879f7763", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "a19f44e0e1a8d24879d641314db2a60742cff32f25c14c69eefbbfbdfd7cfe3e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa469da7-90d1-40d5-81ee-6f2b60e0792f", "node_type": "1", "metadata": {}, "hash": "7ba53e8b46a5a74f990f09411f006cad4726e6002acb19f0a2d65cf95280ab50", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19411, "end_char_idx": 22419, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- They have to arrange each layer so that the output of one layer becomes an input to the other layer.

More recent solutions include:
- Varuna
- Sagemaker

We have not experimented with Varuna and SageMaker but their papers report that they have overcome the list of problems 
mentioned above and that they require smaller changes to the user's model.

Implementations:
- [PyTorch](https://pytorch.org/docs/stable/pipeline.html) (initial support in pytorch-1.8, and progressively getting improved in 1.9 and more so in 1.10). Some [examples](https://github.com/pytorch/pytorch/blob/master/benchmarks/distributed/pipeline/pipe.py)
- [DeepSpeed](https://www.deepspeed.ai/tutorials/pipeline/)
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation - no API.
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.
- [OSLO](https://github.com/tunib-ai/oslo) - this is implemented based on the Hugging Face Transformers.

🤗 Transformers status: as of this writing none of the models supports full-PP. GPT2 and T5 models have naive MP support. 
The main obstacle is being unable to convert the models to `nn.Sequential` and have all the inputs to be Tensors. This 
is because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.

DeepSpeed and Megatron-LM integrations are available in [🤗 Accelerate](https://huggingface.co/docs/accelerate/main/en/usage_guides/deepspeed)

Other approaches:

DeepSpeed, Varuna and SageMaker use the concept of an [Interleaved Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-core-features.html)

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-sagemaker-interleaved-pipeline.png" alt="Interleaved pipeline execution"/>
</div>

Here the bubble (idle time) is further minimized by prioritizing backward passes. Varuna further attempts to improve the 
schedule by using simulations to discover the most efficient scheduling.

OSLO has pipeline parallelism implementation based on the Transformers without `nn.Sequential` conversion.

## Tensor Parallelism

In Tensor Parallelism, each GPU processes a slice of a tensor and only aggregates the full tensor for operations requiring it.
To describe this method, this section of the guide relies on the concepts and diagrams from the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) 
paper: [Efficient Large-Scale Language Model Training on GPU Clusters](https://arxiv.org/abs/2104.04473).

The main building block of any transformer is a fully connected `nn.Linear` followed by a nonlinear activation `GeLU`.
The dot dot-product part of it, following the Megatron's paper notation, can be written as `Y = GeLU(XA)`, where `X` is 
an input vector, `Y` is the output vector, and `A` is the weight matrix.


key: _node_content
string_value:
{"id_": "aa469da7-90d1-40d5-81ee-6f2b60e0792f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b740f795-1b07-405b-9f74-526d876991cd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "e069ddd5bab6d1640b7097a215d5dff5376b313b6458389bb95ebd931c53ba3e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ff262349-3e66-48f6-8fb2-bac27dd7faff", "node_type": "1", "metadata": {}, "hash": "f34bf66b9ce0fd630ec6e95cc081ef3e3b5570135166177ef323ed45223e8d17", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20699, "end_char_idx": 23832, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This 
is because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.

DeepSpeed and Megatron-LM integrations are available in [🤗 Accelerate](https://huggingface.co/docs/accelerate/main/en/usage_guides/deepspeed)

Other approaches:

DeepSpeed, Varuna and SageMaker use the concept of an [Interleaved Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-core-features.html)

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-sagemaker-interleaved-pipeline.png" alt="Interleaved pipeline execution"/>
</div>

Here the bubble (idle time) is further minimized by prioritizing backward passes. Varuna further attempts to improve the 
schedule by using simulations to discover the most efficient scheduling.

OSLO has pipeline parallelism implementation based on the Transformers without `nn.Sequential` conversion.

## Tensor Parallelism

In Tensor Parallelism, each GPU processes a slice of a tensor and only aggregates the full tensor for operations requiring it.
To describe this method, this section of the guide relies on the concepts and diagrams from the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) 
paper: [Efficient Large-Scale Language Model Training on GPU Clusters](https://arxiv.org/abs/2104.04473).

The main building block of any transformer is a fully connected `nn.Linear` followed by a nonlinear activation `GeLU`.
The dot dot-product part of it, following the Megatron's paper notation, can be written as `Y = GeLU(XA)`, where `X` is 
an input vector, `Y` is the output vector, and `A` is the weight matrix.

If we look at the computation in matrix form, you can see how the matrix multiplication can be split between multiple GPUs:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_gemm.png" alt="Parallel GEMM"/>
</div>

If we split the weight matrix `A` column-wise across `N` GPUs and perform matrix multiplications `XA_1` through `XA_n` in parallel, 
then we will end up with `N` output vectors `Y_1, Y_2, ..., Y_n` which can be fed into `GeLU` independently:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-independent-gelu.png" alt="Independent GeLU"/>
</div>

Using this principle, we can update a multi-layer perceptron of arbitrary depth, without the need for any synchronization 
between GPUs until the very end, where we need to reconstruct the output vector from shards. The Megatron-LM paper authors 
provide a helpful illustration for that:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_shard_processing.png" alt="Parallel shard processing"/>
</div>

Parallelizing the multi-headed attention layers is even simpler, since they are already inherently parallel, due to having 
multiple independent heads!


key: _node_content
string_value:
{"id_": "ff262349-3e66-48f6-8fb2-bac27dd7faff", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa469da7-90d1-40d5-81ee-6f2b60e0792f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "d97a9fe91b2e878decbce2bcfc0d7116f08be5854ec9f19898139f3fd62e049a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c9d2c899-a694-4e9a-b32a-c891f774c1ec", "node_type": "1", "metadata": {}, "hash": "1579173a19c4dc268652b0e186f5fbd1bed89637831373171cd6839dfb99a8e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22226, "end_char_idx": 25408, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The dot dot-product part of it, following the Megatron's paper notation, can be written as `Y = GeLU(XA)`, where `X` is 
an input vector, `Y` is the output vector, and `A` is the weight matrix.

If we look at the computation in matrix form, you can see how the matrix multiplication can be split between multiple GPUs:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_gemm.png" alt="Parallel GEMM"/>
</div>

If we split the weight matrix `A` column-wise across `N` GPUs and perform matrix multiplications `XA_1` through `XA_n` in parallel, 
then we will end up with `N` output vectors `Y_1, Y_2, ..., Y_n` which can be fed into `GeLU` independently:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-independent-gelu.png" alt="Independent GeLU"/>
</div>

Using this principle, we can update a multi-layer perceptron of arbitrary depth, without the need for any synchronization 
between GPUs until the very end, where we need to reconstruct the output vector from shards. The Megatron-LM paper authors 
provide a helpful illustration for that:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_shard_processing.png" alt="Parallel shard processing"/>
</div>

Parallelizing the multi-headed attention layers is even simpler, since they are already inherently parallel, due to having 
multiple independent heads!

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_self_attention.png" alt="Parallel self-attention"/>
</div>

Special considerations: TP requires very fast network, and therefore it's not advisable to do TP across more than one node. 
Practically, if a node has 4 GPUs, the highest TP degree is therefore 4. If you need a TP degree of 8, you need to use
nodes that have at least 8 GPUs.

This section is based on the original much more [detailed TP overview](https://github.com/huggingface/transformers/issues/10321#issuecomment-783543530).
by [@anton-l](https://github.com/anton-l).

Alternative names:
- DeepSpeed calls it [tensor slicing](https://www.deepspeed.ai/training/#model-parallelism)

Implementations:
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation, as it's very model-specific
- [parallelformers](https://github.com/tunib-ai/parallelformers) (only inference at the moment)
- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.
- [OSLO](https://github.com/tunib-ai/oslo) has the tensor parallelism implementation based on the Transformers.

SageMaker combines TP with DP for a more efficient processing.

🤗 Transformers status:
- core: not yet implemented in the core
- but if you want inference [parallelformers](https://github.com/tunib-ai/parallelformers) provides this support for most of our models. So until this is implemented in the core you can use theirs.


key: _node_content
string_value:
{"id_": "c9d2c899-a694-4e9a-b32a-c891f774c1ec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ff262349-3e66-48f6-8fb2-bac27dd7faff", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "8558903d67dcc07964f69de962ba029ef82aa05e163ed36a26844674b4a1c49a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "77049925-6721-4279-8dc1-fc55bb1a8fb4", "node_type": "1", "metadata": {}, "hash": "9daaac50d1e5dab161b4257cd75157a8a59554015a1c7a96a3cc7e93ad80c212", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23834, "end_char_idx": 26536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-tp-parallel_self_attention.png" alt="Parallel self-attention"/>
</div>

Special considerations: TP requires very fast network, and therefore it's not advisable to do TP across more than one node. 
Practically, if a node has 4 GPUs, the highest TP degree is therefore 4. If you need a TP degree of 8, you need to use
nodes that have at least 8 GPUs.

This section is based on the original much more [detailed TP overview](https://github.com/huggingface/transformers/issues/10321#issuecomment-783543530).
by [@anton-l](https://github.com/anton-l).

Alternative names:
- DeepSpeed calls it [tensor slicing](https://www.deepspeed.ai/training/#model-parallelism)

Implementations:
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) has an internal implementation, as it's very model-specific
- [parallelformers](https://github.com/tunib-ai/parallelformers) (only inference at the moment)
- [SageMaker](https://arxiv.org/abs/2111.05972) - this is a proprietary solution that can only be used on AWS.
- [OSLO](https://github.com/tunib-ai/oslo) has the tensor parallelism implementation based on the Transformers.

SageMaker combines TP with DP for a more efficient processing.

🤗 Transformers status:
- core: not yet implemented in the core
- but if you want inference [parallelformers](https://github.com/tunib-ai/parallelformers) provides this support for most of our models. So until this is implemented in the core you can use theirs. And hopefully training mode will be supported too.
- Deepspeed-Inference also supports our BERT, GPT-2, and GPT-Neo models in their super-fast CUDA-kernel-based inference mode, see more [here](https://www.deepspeed.ai/tutorials/inference-tutorial/)

🤗 Accelerate integrates with [TP from Megatron-LM](https://huggingface.co/docs/accelerate/v0.23.0/en/usage_guides/megatron_lm).

## Data Parallelism + Pipeline Parallelism

The following diagram from the DeepSpeed [pipeline tutorial](https://www.deepspeed.ai/tutorials/pipeline/) demonstrates 
how one can combine DP with PP.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-zero-dp-pp.png" alt="DP + PP-2d"/>
</div>

Here it's important to see how DP rank 0 doesn't see GPU2 and DP rank 1 doesn't see GPU3. To DP there is just GPUs 0 
and 1 where it feeds data as if there were just 2 GPUs. GPU0 "secretly" offloads some of its load to GPU2 using PP. 
And GPU1 does the same by enlisting GPU3 to its aid.

Since each dimension requires at least 2 GPUs, here you'd need at least 4 GPUs.


key: _node_content
string_value:
{"id_": "77049925-6721-4279-8dc1-fc55bb1a8fb4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c9d2c899-a694-4e9a-b32a-c891f774c1ec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "2402fb110c9e148b965ec754f2bbfbf9395c12d647fa6a39012b04b9e204fa3e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b52bed6b-8f37-407c-b82c-3864a9a7b942", "node_type": "1", "metadata": {}, "hash": "df1b6aab547fc51aab921e9791cf770aab086fc4c079af09e1964ca4389510c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25084, "end_char_idx": 27959, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
SageMaker combines TP with DP for a more efficient processing.

🤗 Transformers status:
- core: not yet implemented in the core
- but if you want inference [parallelformers](https://github.com/tunib-ai/parallelformers) provides this support for most of our models. So until this is implemented in the core you can use theirs. And hopefully training mode will be supported too.
- Deepspeed-Inference also supports our BERT, GPT-2, and GPT-Neo models in their super-fast CUDA-kernel-based inference mode, see more [here](https://www.deepspeed.ai/tutorials/inference-tutorial/)

🤗 Accelerate integrates with [TP from Megatron-LM](https://huggingface.co/docs/accelerate/v0.23.0/en/usage_guides/megatron_lm).

## Data Parallelism + Pipeline Parallelism

The following diagram from the DeepSpeed [pipeline tutorial](https://www.deepspeed.ai/tutorials/pipeline/) demonstrates 
how one can combine DP with PP.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-zero-dp-pp.png" alt="DP + PP-2d"/>
</div>

Here it's important to see how DP rank 0 doesn't see GPU2 and DP rank 1 doesn't see GPU3. To DP there is just GPUs 0 
and 1 where it feeds data as if there were just 2 GPUs. GPU0 "secretly" offloads some of its load to GPU2 using PP. 
And GPU1 does the same by enlisting GPU3 to its aid.

Since each dimension requires at least 2 GPUs, here you'd need at least 4 GPUs.

Implementations:
- [DeepSpeed](https://github.com/microsoft/DeepSpeed)
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972)
- [OSLO](https://github.com/tunib-ai/oslo)

🤗 Transformers status: not yet implemented

## Data Parallelism + Pipeline Parallelism + Tensor Parallelism

To get an even more efficient training a 3D parallelism is used where PP is combined with TP and DP. This can be seen in the following diagram.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-deepspeed-3d.png" alt="dp-pp-tp-3d"/>
</div>

This diagram is from a blog post [3D parallelism: Scaling to trillion-parameter models](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/), which is a good read as well.

Since each dimension requires at least 2 GPUs, here you'd need at least 8 GPUs.

Implementations:
- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - DeepSpeed also includes an even more efficient DP, which they call ZeRO-DP.
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972)
- [OSLO](https://github.com/tunib-ai/oslo)

🤗 Transformers status: not yet implemented, since we have no PP and TP.


key: _node_content
string_value:
{"id_": "b52bed6b-8f37-407c-b82c-3864a9a7b942", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "77049925-6721-4279-8dc1-fc55bb1a8fb4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "b5c7201c4d33f54ecb0953baf9e3efdd912a5a19406cec33fa72a6b55e059ff1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9826968b-2bee-4fbd-9ff3-e91d080fa9e3", "node_type": "1", "metadata": {}, "hash": "798e89309d62d4c9ddb2bb15f18157f28d13b2ac30c4b68483856224c3d5be9e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26538, "end_char_idx": 29515, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Implementations:
- [DeepSpeed](https://github.com/microsoft/DeepSpeed)
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972)
- [OSLO](https://github.com/tunib-ai/oslo)

🤗 Transformers status: not yet implemented

## Data Parallelism + Pipeline Parallelism + Tensor Parallelism

To get an even more efficient training a 3D parallelism is used where PP is combined with TP and DP. This can be seen in the following diagram.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-deepspeed-3d.png" alt="dp-pp-tp-3d"/>
</div>

This diagram is from a blog post [3D parallelism: Scaling to trillion-parameter models](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/), which is a good read as well.

Since each dimension requires at least 2 GPUs, here you'd need at least 8 GPUs.

Implementations:
- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - DeepSpeed also includes an even more efficient DP, which they call ZeRO-DP.
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
- [Varuna](https://github.com/microsoft/varuna)
- [SageMaker](https://arxiv.org/abs/2111.05972)
- [OSLO](https://github.com/tunib-ai/oslo)

🤗 Transformers status: not yet implemented, since we have no PP and TP.

## ZeRO Data Parallelism + Pipeline Parallelism + Tensor Parallelism

One of the main features of DeepSpeed is ZeRO, which is a super-scalable extension of DP. It has already been 
discussed in [ZeRO Data Parallelism](#zero-data-parallelism). Normally it's a standalone feature that doesn't require PP or TP. 
But it can be combined with PP and TP.

When ZeRO-DP is combined with PP (and optionally TP) it typically enables only ZeRO stage 1 (optimizer sharding).

While it's theoretically possible to use ZeRO stage 2 (gradient sharding) with Pipeline Parallelism, it will have negative 
performance impacts. There would need to be an additional reduce-scatter collective for every micro-batch to aggregate 
the gradients before sharding, which adds a potentially significant communication overhead. By nature of Pipeline Parallelism, 
small micro-batches are used and instead the focus is on trying to balance arithmetic intensity (micro-batch size) with
minimizing the Pipeline bubble (number of micro-batches). Therefore those communication costs are going to impact the performance.

In addition, there are already fewer layers than normal due to PP and so the memory savings won't be huge. PP already 
reduces gradient size by ``1/PP``, and so gradient sharding savings on top of that are less significant than pure DP.

ZeRO stage 3 is not a good choice either for the same reason - more inter-node communications required.

And since we have ZeRO, the other benefit is ZeRO-Offload. Since this is stage 1 optimizer states can be offloaded to CPU.


key: _node_content
string_value:
{"id_": "9826968b-2bee-4fbd-9ff3-e91d080fa9e3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b52bed6b-8f37-407c-b82c-3864a9a7b942", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "7c5167ef9a5da1263d4449bcbb2850224245c015cb3dc0b7be9057afd754b2cf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "48a04f83-c359-4608-8401-dd7c1d8e9bbf", "node_type": "1", "metadata": {}, "hash": "44d101216853fa6795481b8a4b9604ed5895bc73e6c5de3023a821484adedad3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27961, "end_char_idx": 31196, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## ZeRO Data Parallelism + Pipeline Parallelism + Tensor Parallelism

One of the main features of DeepSpeed is ZeRO, which is a super-scalable extension of DP. It has already been 
discussed in [ZeRO Data Parallelism](#zero-data-parallelism). Normally it's a standalone feature that doesn't require PP or TP. 
But it can be combined with PP and TP.

When ZeRO-DP is combined with PP (and optionally TP) it typically enables only ZeRO stage 1 (optimizer sharding).

While it's theoretically possible to use ZeRO stage 2 (gradient sharding) with Pipeline Parallelism, it will have negative 
performance impacts. There would need to be an additional reduce-scatter collective for every micro-batch to aggregate 
the gradients before sharding, which adds a potentially significant communication overhead. By nature of Pipeline Parallelism, 
small micro-batches are used and instead the focus is on trying to balance arithmetic intensity (micro-batch size) with
minimizing the Pipeline bubble (number of micro-batches). Therefore those communication costs are going to impact the performance.

In addition, there are already fewer layers than normal due to PP and so the memory savings won't be huge. PP already 
reduces gradient size by ``1/PP``, and so gradient sharding savings on top of that are less significant than pure DP.

ZeRO stage 3 is not a good choice either for the same reason - more inter-node communications required.

And since we have ZeRO, the other benefit is ZeRO-Offload. Since this is stage 1 optimizer states can be offloaded to CPU.

Implementations:
- [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed) and [Megatron-Deepspeed from BigScience](https://github.com/bigscience-workshop/Megatron-DeepSpeed), which is the fork of the former repo.
- [OSLO](https://github.com/tunib-ai/oslo)

Important papers:

- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](
https://arxiv.org/abs/2201.11990)

🤗 Transformers status: not yet implemented, since we have no PP and TP.

## FlexFlow

[FlexFlow](https://github.com/flexflow/FlexFlow) also solves the parallelization problem in a slightly different approach.

Paper: ["Beyond Data and Model Parallelism for Deep Neural Networks" by Zhihao Jia, Matei Zaharia, Alex Aiken](https://arxiv.org/abs/1807.05358)

It performs a sort of 4D Parallelism over Sample-Operator-Attribute-Parameter.

1. Sample = Data Parallelism (sample-wise parallel)
2. Operator = Parallelize a single operation into several sub-operations
3. Attribute = Data Parallelism (length-wise parallel)
4. Parameter = Model Parallelism (regardless of dimension - horizontal or vertical)

Examples:
* Sample

Let's take 10 batches of sequence length 512. If we parallelize them by sample dimension into 2 devices, we get 10 x 512 which becomes be 5 x 2 x 512.

* Operator

If we perform layer normalization, we compute std first and mean second, and then we can normalize data. 
Operator parallelism allows computing std and mean in parallel. So if we parallelize them by operator dimension into 2 
devices (cuda:0, cuda:1), first we copy input data into both devices, and cuda:0 computes std, cuda:1 computes mean at the same time.


key: _node_content
string_value:
{"id_": "48a04f83-c359-4608-8401-dd7c1d8e9bbf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9826968b-2bee-4fbd-9ff3-e91d080fa9e3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "ca19081580df4c369c168f64b64631d690664a08eba965819c02ca0febbef29a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea6113b7-3194-4efb-ae96-ff85647df9d2", "node_type": "1", "metadata": {}, "hash": "464d90f681f30c534c12c07ead6a8428b748112b55decfe75eb4c0b554638191", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29745, "end_char_idx": 32980, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [OSLO](https://github.com/tunib-ai/oslo)

Important papers:

- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](
https://arxiv.org/abs/2201.11990)

🤗 Transformers status: not yet implemented, since we have no PP and TP.

## FlexFlow

[FlexFlow](https://github.com/flexflow/FlexFlow) also solves the parallelization problem in a slightly different approach.

Paper: ["Beyond Data and Model Parallelism for Deep Neural Networks" by Zhihao Jia, Matei Zaharia, Alex Aiken](https://arxiv.org/abs/1807.05358)

It performs a sort of 4D Parallelism over Sample-Operator-Attribute-Parameter.

1. Sample = Data Parallelism (sample-wise parallel)
2. Operator = Parallelize a single operation into several sub-operations
3. Attribute = Data Parallelism (length-wise parallel)
4. Parameter = Model Parallelism (regardless of dimension - horizontal or vertical)

Examples:
* Sample

Let's take 10 batches of sequence length 512. If we parallelize them by sample dimension into 2 devices, we get 10 x 512 which becomes be 5 x 2 x 512.

* Operator

If we perform layer normalization, we compute std first and mean second, and then we can normalize data. 
Operator parallelism allows computing std and mean in parallel. So if we parallelize them by operator dimension into 2 
devices (cuda:0, cuda:1), first we copy input data into both devices, and cuda:0 computes std, cuda:1 computes mean at the same time.

* Attribute

We have 10 batches of 512 length. If we parallelize them by attribute dimension into 2 devices, 10 x 512 will be 10 x 2 x 256.

* Parameter

It is similar with tensor model parallelism or naive layer-wise model parallelism.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-flexflow.jpeg" alt="flex-flow-soap"/>
</div>

The significance of this framework is that it takes resources like (1) GPU/TPU/CPU vs. (2) RAM/DRAM vs. (3) 
fast-intra-connect/slow-inter-connect and it automatically optimizes all these algorithmically deciding which 
parallelisation to use where.

One very important aspect is that FlexFlow is designed for optimizing DNN parallelizations for models with static and 
fixed workloads, since models with dynamic behavior may prefer different parallelization strategies across iterations.

So the promise is very attractive - it runs a 30min simulation on the cluster of choice and it comes up with the best 
strategy to utilise this specific environment. If you add/remove/replace any parts it'll run and re-optimize the plan 
for that. And then you can train. A different setup will have its own custom optimization.

🤗 Transformers status: Transformers models are FX-trace-able via [transformers.utils.fx](https://github.com/huggingface/transformers/blob/master/src/transformers/utils/fx.py), 
which is a prerequisite for FlexFlow, however, changes are required on the FlexFlow side to make it work with Transformers models.

## GPU selection

When training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first.


key: _node_content
string_value:
{"id_": "ea6113b7-3194-4efb-ae96-ff85647df9d2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "48a04f83-c359-4608-8401-dd7c1d8e9bbf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "36e87959957cbc663eb512609fae1a0a1bdd8589b93c75eaf3a0cff0bc8a1589", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8896779a-3f18-425b-905b-208e5458bad9", "node_type": "1", "metadata": {}, "hash": "ec9e668809547ed02db8f27a69bfa8e817b496b9f5b05eadebe77e201862218f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31245, "end_char_idx": 34248, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If we parallelize them by attribute dimension into 2 devices, 10 x 512 will be 10 x 2 x 256.

* Parameter

It is similar with tensor model parallelism or naive layer-wise model parallelism.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/parallelism-flexflow.jpeg" alt="flex-flow-soap"/>
</div>

The significance of this framework is that it takes resources like (1) GPU/TPU/CPU vs. (2) RAM/DRAM vs. (3) 
fast-intra-connect/slow-inter-connect and it automatically optimizes all these algorithmically deciding which 
parallelisation to use where.

One very important aspect is that FlexFlow is designed for optimizing DNN parallelizations for models with static and 
fixed workloads, since models with dynamic behavior may prefer different parallelization strategies across iterations.

So the promise is very attractive - it runs a 30min simulation on the cluster of choice and it comes up with the best 
strategy to utilise this specific environment. If you add/remove/replace any parts it'll run and re-optimize the plan 
for that. And then you can train. A different setup will have its own custom optimization.

🤗 Transformers status: Transformers models are FX-trace-able via [transformers.utils.fx](https://github.com/huggingface/transformers/blob/master/src/transformers/utils/fx.py), 
which is a prerequisite for FlexFlow, however, changes are required on the FlexFlow side to make it work with Transformers models.

## GPU selection

When training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first. The selection process works for both [DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) and [DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html) to use only a subset of the available GPUs, and you don't need Accelerate or the [DeepSpeed integration](./main_classes/deepspeed).

### Number of GPUs

For example, if you have 4 GPUs and you only want to use the first 2:

<hfoptions id="select-gpu">
<hfoption id="torchrun">

Use the `--nproc_per_node` to select how many GPUs to use.

```bash
torchrun --nproc_per_node=2  trainer-program.py ...
```

</hfoption>
<hfoption id="Accelerate">

Use `--num_processes` to select how many GPUs to use.

```bash
accelerate launch --num_processes 2 trainer-program.py ...
```

</hfoption>
<hfoption id="DeepSpeed">

Use `--num_gpus` to select how many GPUs to use.

```bash
deepspeed --num_gpus 2 trainer-program.py ...
```

</hfoption>
</hfoptions>

### Order of GPUs

Now, to select which GPUs to use and their order, you'll use the `CUDA_VISIBLE_DEVICES` environment variable. It is easiest to set the environment variable in a `~/bashrc` or another startup config file. `CUDA_VISIBLE_DEVICES` is used to map which GPUs are used.


key: _node_content
string_value:
{"id_": "8896779a-3f18-425b-905b-208e5458bad9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea6113b7-3194-4efb-ae96-ff85647df9d2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "286fff1222eeb809215f173368f63d99883d85ab23e65ea2f9f517adb6a25428", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c7171c25-753f-467b-ba58-d4907c4da862", "node_type": "1", "metadata": {}, "hash": "d9887dcaa861f52c59264cf00383a0d11b00c25a3b1f749e63a7580a63931f94", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32750, "end_char_idx": 35762, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## GPU selection

When training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first. The selection process works for both [DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) and [DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html) to use only a subset of the available GPUs, and you don't need Accelerate or the [DeepSpeed integration](./main_classes/deepspeed).

### Number of GPUs

For example, if you have 4 GPUs and you only want to use the first 2:

<hfoptions id="select-gpu">
<hfoption id="torchrun">

Use the `--nproc_per_node` to select how many GPUs to use.

```bash
torchrun --nproc_per_node=2  trainer-program.py ...
```

</hfoption>
<hfoption id="Accelerate">

Use `--num_processes` to select how many GPUs to use.

```bash
accelerate launch --num_processes 2 trainer-program.py ...
```

</hfoption>
<hfoption id="DeepSpeed">

Use `--num_gpus` to select how many GPUs to use.

```bash
deepspeed --num_gpus 2 trainer-program.py ...
```

</hfoption>
</hfoptions>

### Order of GPUs

Now, to select which GPUs to use and their order, you'll use the `CUDA_VISIBLE_DEVICES` environment variable. It is easiest to set the environment variable in a `~/bashrc` or another startup config file. `CUDA_VISIBLE_DEVICES` is used to map which GPUs are used. For example, if you have 4 GPUs (0, 1, 2, 3) and you only want to run GPUs 0 and 2:

```bash
CUDA_VISIBLE_DEVICES=0,2 torchrun trainer-program.py ...
```

Only the 2 physical GPUs (0 and 2) are "visible" to PyTorch and these are mapped to `cuda:0` and `cuda:1` respectively. You can also reverse the order of the GPUs to use 2 first. Now, the mapping is `cuda:1` for GPU 0 and `cuda:0` for GPU 2.

```bash
CUDA_VISIBLE_DEVICES=2,0 torchrun trainer-program.py ...
```

You can also set the `CUDA_VISIBLE_DEVICES` environment variable to an empty value to create an environment without GPUs.

```bash
CUDA_VISIBLE_DEVICES= python trainer-program.py ...
```

<Tip warning={true}>

As with any environment variable, they can be exported instead of being added to the command line. However, this is not recommended because it can be confusing if you forget how the environment variable was setup and you end up using the wrong GPUs. Instead, it is common practice to set the environment variable for a specific training run on the same command line.

</Tip>

`CUDA_DEVICE_ORDER` is an alternative environment variable you can use to control how the GPUs are ordered. You can either order them by:

1. PCIe bus ID's that matches the order of [`nvidia-smi`](https://developer.nvidia.com/nvidia-system-management-interface) and [`rocm-smi`](https://rocm.docs.amd.com/projects/rocm_smi_lib/en/latest/.doxygen/docBin/html/index.html) for NVIDIA and AMD GPUs respectively

```bash
export CUDA_DEVICE_ORDER=PCI_BUS_ID
```

2.


key: _node_content
string_value:
{"id_": "c7171c25-753f-467b-ba58-d4907c4da862", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8896779a-3f18-425b-905b-208e5458bad9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers"}, "hash": "f19c9d3c93d37e938a5569698ab3b47d1084f3f9c6ba8910b2247aee442c86f2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34249, "end_char_idx": 36278, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, if you have 4 GPUs (0, 1, 2, 3) and you only want to run GPUs 0 and 2:

```bash
CUDA_VISIBLE_DEVICES=0,2 torchrun trainer-program.py ...
```

Only the 2 physical GPUs (0 and 2) are "visible" to PyTorch and these are mapped to `cuda:0` and `cuda:1` respectively. You can also reverse the order of the GPUs to use 2 first. Now, the mapping is `cuda:1` for GPU 0 and `cuda:0` for GPU 2.

```bash
CUDA_VISIBLE_DEVICES=2,0 torchrun trainer-program.py ...
```

You can also set the `CUDA_VISIBLE_DEVICES` environment variable to an empty value to create an environment without GPUs.

```bash
CUDA_VISIBLE_DEVICES= python trainer-program.py ...
```

<Tip warning={true}>

As with any environment variable, they can be exported instead of being added to the command line. However, this is not recommended because it can be confusing if you forget how the environment variable was setup and you end up using the wrong GPUs. Instead, it is common practice to set the environment variable for a specific training run on the same command line.

</Tip>

`CUDA_DEVICE_ORDER` is an alternative environment variable you can use to control how the GPUs are ordered. You can either order them by:

1. PCIe bus ID's that matches the order of [`nvidia-smi`](https://developer.nvidia.com/nvidia-system-management-interface) and [`rocm-smi`](https://rocm.docs.amd.com/projects/rocm_smi_lib/en/latest/.doxygen/docBin/html/index.html) for NVIDIA and AMD GPUs respectively

```bash
export CUDA_DEVICE_ORDER=PCI_BUS_ID
```

2. GPU compute ability

```bash
export CUDA_DEVICE_ORDER=FASTEST_FIRST
```

The `CUDA_DEVICE_ORDER` is especially useful if your training setup consists of an older and newer GPU, where the older GPU appears first, but you cannot physically swap the cards to make the newer GPU appear first. In this case, set `CUDA_DEVICE_ORDER=FASTEST_FIRST` to always use the newer and faster GPU first (`nvidia-smi` or `rocm-smi` still reports the GPUs in their PCIe order). Or you could also set `export CUDA_VISIBLE_DEVICES=1,0`.


key: _node_content
string_value:
{"id_": "ddb9a2b0-da44-48ec-8f43-b5ce2ef18918", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "54374f90-6c4a-4a8a-8db6-43b2111b020d", "node_type": "1", "metadata": {}, "hash": "aaf2f9718a6a3c761152d24cd7272130958591333fb5cb963ccc19c90d0ddb3b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3074, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Templates for Chat Models

## Introduction

An increasingly common use case for LLMs is **chat**. In a chat context, rather than continuing a single string
of text (as is the case with a standard language model), the model instead continues a conversation that consists
of one or more **messages**, each of which includes a **role**, like "user" or "assistant", as well as message text.

Much like tokenization, different models expect very different input formats for chat. This is the reason we added
**chat templates** as a feature. Chat templates are part of the tokenizer. They specify how to convert conversations, 
represented as lists of messages, into a single tokenizable string in the format that the model expects. 

Let's make this concrete with a quick example using the `BlenderBot` model. BlenderBot has an extremely simple default 
template, which mostly just adds whitespace between rounds of dialogue:

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")

>>> chat = [
...    {"role": "user", "content": "Hello, how are you?"},
...    {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
...    {"role": "user", "content": "I'd like to show off how chat templating works!"},
... ]

>>> tokenizer.apply_chat_template(chat, tokenize=False)
" Hello, how are you?  I'm doing great. How can I help you today?   I'd like to show off how chat templating works!</s>"
```

Notice how the entire chat is condensed into a single string. If we use `tokenize=True`, which is the default setting,
that string will also be tokenized for us. To see a more complex template in action, though, let's use the 
`mistralai/Mistral-7B-Instruct-v0.1` model.

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")

>>> chat = [
...   {"role": "user", "content": "Hello, how are you?"},
...   {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
...   {"role": "user", "content": "I'd like to show off how chat templating works!"},
... ]

>>> tokenizer.apply_chat_template(chat, tokenize=False)
"<s>[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]"
```

Note that this time, the tokenizer has added the control tokens [INST] and [/INST] to indicate the start and end of 
user messages (but not assistant messages!). Mistral-instruct was trained with these tokens, but BlenderBot was not.

## How do I use chat templates?

As you can see in the example above, chat templates are easy to use. Simply build a list of messages, with `role`
and `content` keys, and then pass it to the [`~PreTrainedTokenizer.apply_chat_template`] method. Once you do that,
you'll get output that's ready to go! When using chat templates as input for model generation, it's also a good idea
to use `add_generation_prompt=True` to add a [generation prompt](#what-are-generation-prompts).


key: _node_content
string_value:
{"id_": "54374f90-6c4a-4a8a-8db6-43b2111b020d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ddb9a2b0-da44-48ec-8f43-b5ce2ef18918", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "1d277dccbd88ca7d48b0da5d8e53973f43876e0bce175cb78c7da90b33238930", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "027683d4-8f1c-4607-90d7-056e9efc3a3f", "node_type": "1", "metadata": {}, "hash": "e25898941bcd9e254bdf0854b82d2f2b42af8802a836f50eca862bc3e83d739b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1657, "end_char_idx": 4667, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To see a more complex template in action, though, let's use the 
`mistralai/Mistral-7B-Instruct-v0.1` model.

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")

>>> chat = [
...   {"role": "user", "content": "Hello, how are you?"},
...   {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
...   {"role": "user", "content": "I'd like to show off how chat templating works!"},
... ]

>>> tokenizer.apply_chat_template(chat, tokenize=False)
"<s>[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]"
```

Note that this time, the tokenizer has added the control tokens [INST] and [/INST] to indicate the start and end of 
user messages (but not assistant messages!). Mistral-instruct was trained with these tokens, but BlenderBot was not.

## How do I use chat templates?

As you can see in the example above, chat templates are easy to use. Simply build a list of messages, with `role`
and `content` keys, and then pass it to the [`~PreTrainedTokenizer.apply_chat_template`] method. Once you do that,
you'll get output that's ready to go! When using chat templates as input for model generation, it's also a good idea
to use `add_generation_prompt=True` to add a [generation prompt](#what-are-generation-prompts). 

Here's an example of preparing input for `model.generate()`, using the `Zephyr` assistant model:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "HuggingFaceH4/zephyr-7b-beta"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)  # You may want to use bfloat16 and/or move to GPU here

messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
 ]
tokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
print(tokenizer.decode(tokenized_chat[0]))
```
This will yield a string in the input format that Zephyr expects. 
```text
<|system|>
You are a friendly chatbot who always responds in the style of a pirate</s> 
<|user|>
How many helicopters can a human eat in one sitting?</s> 
<|assistant|>
```

Now that our input is formatted correctly for Zephyr, we can use the model to generate a response to the user's question:

```python
outputs = model.generate(tokenized_chat, max_new_tokens=128) 
print(tokenizer.decode(outputs[0]))
```

This will yield:

```text
<|system|>
You are a friendly chatbot who always responds in the style of a pirate</s> 
<|user|>
How many helicopters can a human eat in one sitting?</s> 
<|assistant|>
Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines.


key: _node_content
string_value:
{"id_": "027683d4-8f1c-4607-90d7-056e9efc3a3f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "54374f90-6c4a-4a8a-8db6-43b2111b020d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "15655ea7b342721aa2b8a6168f703ea5a7b97a78cdff0ff043779cc5e0258263", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a94fee9d-0f1f-4032-8f09-5fa3510ab848", "node_type": "1", "metadata": {}, "hash": "f9f544acf6838481cccf8e0cb3f215a2b7dd1199c83fd4f9a26b4699fe3f0e14", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3077, "end_char_idx": 6139, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here's an example of preparing input for `model.generate()`, using the `Zephyr` assistant model:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "HuggingFaceH4/zephyr-7b-beta"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)  # You may want to use bfloat16 and/or move to GPU here

messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
 ]
tokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
print(tokenizer.decode(tokenized_chat[0]))
```
This will yield a string in the input format that Zephyr expects. 
```text
<|system|>
You are a friendly chatbot who always responds in the style of a pirate</s> 
<|user|>
How many helicopters can a human eat in one sitting?</s> 
<|assistant|>
```

Now that our input is formatted correctly for Zephyr, we can use the model to generate a response to the user's question:

```python
outputs = model.generate(tokenized_chat, max_new_tokens=128) 
print(tokenizer.decode(outputs[0]))
```

This will yield:

```text
<|system|>
You are a friendly chatbot who always responds in the style of a pirate</s> 
<|user|>
How many helicopters can a human eat in one sitting?</s> 
<|assistant|>
Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all.
```

Arr, 'twas easy after all!

## Is there an automated pipeline for chat?

Yes, there is! Our text generation pipelines support chat inputs, which makes it easy to use chat models. In the past,
we used to use a dedicated "ConversationalPipeline" class, but this has now been deprecated and its functionality
has been merged into the [`TextGenerationPipeline`]. Let's try the `Zephyr` example again, but this time using 
a pipeline:

```python
from transformers import pipeline

pipe = pipeline("text-generation", "HuggingFaceH4/zephyr-7b-beta")
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
print(pipe(messages, max_new_tokens=128)[0]['generated_text'][-1])  # Print the assistant's response
```

```text
{'role': 'assistant', 'content': "Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'.


key: _node_content
string_value:
{"id_": "a94fee9d-0f1f-4032-8f09-5fa3510ab848", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "027683d4-8f1c-4607-90d7-056e9efc3a3f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "b60584dc5b44c40d2a48eb8f1a82e2b23facc385e05070881257d6f34dbb77e1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "320802b0-03b6-4a74-ae00-0ca9b36a5d8a", "node_type": "1", "metadata": {}, "hash": "b7f27af8fe5a32cfaa70506fe28a9d578d588fb697a2d7c25045332f6db5780b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4616, "end_char_idx": 6867, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all.
```

Arr, 'twas easy after all!

## Is there an automated pipeline for chat?

Yes, there is! Our text generation pipelines support chat inputs, which makes it easy to use chat models. In the past,
we used to use a dedicated "ConversationalPipeline" class, but this has now been deprecated and its functionality
has been merged into the [`TextGenerationPipeline`]. Let's try the `Zephyr` example again, but this time using 
a pipeline:

```python
from transformers import pipeline

pipe = pipeline("text-generation", "HuggingFaceH4/zephyr-7b-beta")
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
print(pipe(messages, max_new_tokens=128)[0]['generated_text'][-1])  # Print the assistant's response
```

```text
{'role': 'assistant', 'content': "Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all."}
```

The pipeline will take care of all the details of tokenization and calling `apply_chat_template` for you -
once the model has a chat template, all you need to do is initialize the pipeline and pass it the list of messages!

## What are "generation prompts"?

You may have noticed that the `apply_chat_template` method has an `add_generation_prompt` argument. This argument tells
the template to add tokens that indicate the start of a bot response. For example, consider the following chat:

```python
messages = [
    {"role": "user", "content": "Hi there!"},
    {"role": "assistant", "content": "Nice to meet you!"},
    {"role": "user", "content": "Can I ask a question?"}


key: _node_content
string_value:
{"id_": "320802b0-03b6-4a74-ae00-0ca9b36a5d8a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a94fee9d-0f1f-4032-8f09-5fa3510ab848", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "2b5ec3881d0222ea9fa2886ec58cd52d111b7b475ae795f5087d4245ed978dfc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "faf0c33d-6929-49b6-8c6b-f302dd41066e", "node_type": "1", "metadata": {}, "hash": "f0649922fc5861e201401aeab73c8d44558eb136d0cb43825cb924faad9f89ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5677, "end_char_idx": 8829, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
]
print(pipe(messages, max_new_tokens=128)[0]['generated_text'][-1])  # Print the assistant's response
```

```text
{'role': 'assistant', 'content': "Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all."}
```

The pipeline will take care of all the details of tokenization and calling `apply_chat_template` for you -
once the model has a chat template, all you need to do is initialize the pipeline and pass it the list of messages!

## What are "generation prompts"?

You may have noticed that the `apply_chat_template` method has an `add_generation_prompt` argument. This argument tells
the template to add tokens that indicate the start of a bot response. For example, consider the following chat:

```python
messages = [
    {"role": "user", "content": "Hi there!"},
    {"role": "assistant", "content": "Nice to meet you!"},
    {"role": "user", "content": "Can I ask a question?"}
]
```

Here's what this will look like without a generation prompt, using the ChatML template we saw in the Zephyr example:

```python
tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
"""<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
"""
```

And here's what it looks like **with** a generation prompt:

```python
tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
"""<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant
"""
```

Note that this time, we've added the tokens that indicate the start of a bot response. This ensures that when the model
generates text it will write a bot response instead of doing something unexpected, like continuing the user's 
message. Remember, chat models are still just language models - they're trained to continue text, and chat is just a 
special kind of text to them! You need to guide them with appropriate control tokens, so they know what they're 
supposed to be doing.

Not all models require generation prompts. Some models, like BlenderBot and LLaMA, don't have any
special tokens before bot responses. In these cases, the `add_generation_prompt` argument will have no effect. The exact
effect that `add_generation_prompt` has will depend on the template being used.

## Can I use chat templates in training?

Yes! This is a good way to ensure that the chat template matches the tokens the model sees during training.
We recommend that you apply the chat template as a preprocessing step for your dataset. After this, you
can simply continue like any other language model training task. When training, you should usually set 
`add_generation_prompt=False`, because the added tokens to prompt an assistant response will not be helpful during 
training.


key: _node_content
string_value:
{"id_": "faf0c33d-6929-49b6-8c6b-f302dd41066e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "320802b0-03b6-4a74-ae00-0ca9b36a5d8a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8d46a33000306c2fa7627e4fc929570a9288c3b1569c82172c5577503d63afa2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a04f2683-e3ef-43c1-8b6f-81a07bb9051d", "node_type": "1", "metadata": {}, "hash": "f08c3ff1288ed11d4fdabe5de208e5e1203b3d1df44d1cc93033b09803551eb2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7648, "end_char_idx": 11113, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This ensures that when the model
generates text it will write a bot response instead of doing something unexpected, like continuing the user's 
message. Remember, chat models are still just language models - they're trained to continue text, and chat is just a 
special kind of text to them! You need to guide them with appropriate control tokens, so they know what they're 
supposed to be doing.

Not all models require generation prompts. Some models, like BlenderBot and LLaMA, don't have any
special tokens before bot responses. In these cases, the `add_generation_prompt` argument will have no effect. The exact
effect that `add_generation_prompt` has will depend on the template being used.

## Can I use chat templates in training?

Yes! This is a good way to ensure that the chat template matches the tokens the model sees during training.
We recommend that you apply the chat template as a preprocessing step for your dataset. After this, you
can simply continue like any other language model training task. When training, you should usually set 
`add_generation_prompt=False`, because the added tokens to prompt an assistant response will not be helpful during 
training. Let's see an example:

```python
from transformers import AutoTokenizer
from datasets import Dataset

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")

chat1 = [
    {"role": "user", "content": "Which is bigger, the moon or the sun?"},
    {"role": "assistant", "content": "The sun."}
]
chat2 = [
    {"role": "user", "content": "Which is bigger, a virus or a bacterium?"},
    {"role": "assistant", "content": "A bacterium."}
]

dataset = Dataset.from_dict({"chat": [chat1, chat2]})
dataset = dataset.map(lambda x: {"formatted_chat": tokenizer.apply_chat_template(x["chat"], tokenize=False, add_generation_prompt=False)})
print(dataset['formatted_chat'][0])
```
And we get:
```text
<|user|>
Which is bigger, the moon or the sun?</s>
<|assistant|>
The sun.</s>
```

From here, just continue training like you would with a standard language modelling task, using the `formatted_chat` column.

<Tip>
If you format text with `apply_chat_template(tokenize=False)` and then tokenize it in a separate step, you should set the argument
`add_special_tokens=False`. If you use `apply_chat_template(tokenize=True)`, you don't need to worry about this!

By default, some tokenizers add special tokens like `<bos>` and `<eos>` to text they tokenize. Chat templates should 
always include all of the special tokens they need, and so adding extra special tokens with
the default `add_special_tokens=True` can result in incorrect or duplicated special tokens, which will hurt model
performance.
</Tip>

## Advanced: Extra inputs to chat templates

The only argument that `apply_chat_template` requires is `messages`. However, you can pass any keyword
argument to `apply_chat_template` and it will be accessible inside the template. This gives you a lot of freedom to use
chat templates for many things. There are no restrictions on the names or the format of these arguments - you can pass
strings, lists, dicts or whatever else you want. 

That said, there are some common use-cases for these extra arguments,
such as passing tools for function calling, or documents for retrieval-augmented generation. In these common cases,
we have some opinionated recommendations about what the names and formats of these arguments should be, which are
described in the sections below.


key: _node_content
string_value:
{"id_": "a04f2683-e3ef-43c1-8b6f-81a07bb9051d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "faf0c33d-6929-49b6-8c6b-f302dd41066e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8911e65bf29946376b8191e029838d0ca16609ef84ef53a2784f8033488f0cc0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "968ef10a-9063-48e1-a5c4-515679695989", "node_type": "1", "metadata": {}, "hash": "078d3159ccb378f9113bf75b886b87054273edcb8a4d98ec3596a8194a692e90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9747, "end_char_idx": 13181, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>
If you format text with `apply_chat_template(tokenize=False)` and then tokenize it in a separate step, you should set the argument
`add_special_tokens=False`. If you use `apply_chat_template(tokenize=True)`, you don't need to worry about this!

By default, some tokenizers add special tokens like `<bos>` and `<eos>` to text they tokenize. Chat templates should 
always include all of the special tokens they need, and so adding extra special tokens with
the default `add_special_tokens=True` can result in incorrect or duplicated special tokens, which will hurt model
performance.
</Tip>

## Advanced: Extra inputs to chat templates

The only argument that `apply_chat_template` requires is `messages`. However, you can pass any keyword
argument to `apply_chat_template` and it will be accessible inside the template. This gives you a lot of freedom to use
chat templates for many things. There are no restrictions on the names or the format of these arguments - you can pass
strings, lists, dicts or whatever else you want. 

That said, there are some common use-cases for these extra arguments,
such as passing tools for function calling, or documents for retrieval-augmented generation. In these common cases,
we have some opinionated recommendations about what the names and formats of these arguments should be, which are
described in the sections below. We encourage model authors to make their chat templates compatible with this format,
to make it easy to transfer tool-calling code between models.

## Advanced: Tool use / function calling

"Tool use" LLMs can choose to call functions as external tools before generating an answer. When passing tools
to a tool-use model, you can simply pass a list of functions to the `tools` argument:

```python
import datetime

def current_time():
    """Get the current local time as a string."""
    return str(datetime.now())

def multiply(a: float, b: float):
    """
    A function that multiplies two numbers
    
    Args:
        a: The first number to multiply
        b: The second number to multiply
    """
    return a * b

tools = [current_time, multiply]

model_input = tokenizer.apply_chat_template(
    messages,
    tools=tools
)
```

In order for this to work correctly, you should write your functions in the format above, so that they can be parsed
correctly as tools. Specifically, you should follow these rules:

- The function should have a descriptive name
- Every argument must have a type hint
- The function must have a docstring in the standard Google style (in other words, an initial function description  
  followed by an `Args:` block that describes the arguments, unless the function does not have any arguments. 
- Do not include types in the `Args:` block. In other words, write `a: The first number to multiply`, not
  `a (int): The first number to multiply`. Type hints should go in the function header instead.
- The function can have a return type and a `Returns:` block in the docstring. However, these are optional
  because most tool-use models ignore them.

### Passing tool results to the model

The sample code above is enough to list the available tools for your model, but what happens if it wants to actually use
one? If that happens, you should:

1. Parse the model's output to get the tool name(s) and arguments.
2. Add the model's tool call(s) to the conversation.
3. Call the corresponding function(s) with those arguments.
4.


key: _node_content
string_value:
{"id_": "968ef10a-9063-48e1-a5c4-515679695989", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a04f2683-e3ef-43c1-8b6f-81a07bb9051d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "5c02aa5cbf37f2b03e31a69067e7b8041a342cf3bd006a4f3bf03bb7fc777de6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ec746d55-c0f6-419e-b8c9-b174ad8c087c", "node_type": "1", "metadata": {}, "hash": "d78e5b774a9839cde82bd2dae5430ad22c6bbc77bcfa035ce773a51d3b34ea5f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11603, "end_char_idx": 14801, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
return str(datetime.now())

def multiply(a: float, b: float):
    """
    A function that multiplies two numbers
    
    Args:
        a: The first number to multiply
        b: The second number to multiply
    """
    return a * b

tools = [current_time, multiply]

model_input = tokenizer.apply_chat_template(
    messages,
    tools=tools
)
```

In order for this to work correctly, you should write your functions in the format above, so that they can be parsed
correctly as tools. Specifically, you should follow these rules:

- The function should have a descriptive name
- Every argument must have a type hint
- The function must have a docstring in the standard Google style (in other words, an initial function description  
  followed by an `Args:` block that describes the arguments, unless the function does not have any arguments. 
- Do not include types in the `Args:` block. In other words, write `a: The first number to multiply`, not
  `a (int): The first number to multiply`. Type hints should go in the function header instead.
- The function can have a return type and a `Returns:` block in the docstring. However, these are optional
  because most tool-use models ignore them.

### Passing tool results to the model

The sample code above is enough to list the available tools for your model, but what happens if it wants to actually use
one? If that happens, you should:

1. Parse the model's output to get the tool name(s) and arguments.
2. Add the model's tool call(s) to the conversation.
3. Call the corresponding function(s) with those arguments.
4. Add the result(s) to the conversation

### A complete tool use example

Let's walk through a tool use example, step by step. For this example, we will use an 8B `Hermes-2-Pro` model,
as it is one of the highest-performing tool-use models in its size category at the time of writing. If you have the
memory, you can consider using a larger model instead like [Command-R](https://huggingface.co/CohereForAI/c4ai-command-r-v01)
or [Mixtral-8x22B](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), both of which also support tool use
and offer even stronger performance.

First, let's load our model and tokenizer:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "NousResearch/Hermes-2-Pro-Llama-3-8B"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, revision="pr/13")
model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto")
```

Next, let's define a list of tools:

```python
def get_current_temperature(location: str, unit: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
        unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
    Returns:
        The current temperature at the specified location in the specified units, as a float.
    """
    return 22.  # A real function should probably actually get the temperature!

def get_current_wind_speed(location: str) -> float:
    """
    Get the current wind speed in km/h at a given location.


key: _node_content
string_value:
{"id_": "ec746d55-c0f6-419e-b8c9-b174ad8c087c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "968ef10a-9063-48e1-a5c4-515679695989", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "10278b1e9fb039aa5901b75a910a9def302bec7e499b509bf8ce42d10b53fdfb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b0a18978-a762-4bb0-baaf-517897c124f7", "node_type": "1", "metadata": {}, "hash": "a7bc31e4dbe6b809646f0e3d0fa24d48d8bda94641c720dc3f94d18cfc4b9827", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13307, "end_char_idx": 16415, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For this example, we will use an 8B `Hermes-2-Pro` model,
as it is one of the highest-performing tool-use models in its size category at the time of writing. If you have the
memory, you can consider using a larger model instead like [Command-R](https://huggingface.co/CohereForAI/c4ai-command-r-v01)
or [Mixtral-8x22B](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), both of which also support tool use
and offer even stronger performance.

First, let's load our model and tokenizer:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "NousResearch/Hermes-2-Pro-Llama-3-8B"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, revision="pr/13")
model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto")
```

Next, let's define a list of tools:

```python
def get_current_temperature(location: str, unit: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
        unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
    Returns:
        The current temperature at the specified location in the specified units, as a float.
    """
    return 22.  # A real function should probably actually get the temperature!

def get_current_wind_speed(location: str) -> float:
    """
    Get the current wind speed in km/h at a given location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current wind speed at the given location in km/h, as a float.
    """
    return 6.  # A real function should probably actually get the wind speed!

tools = [get_current_temperature, get_current_wind_speed]
```

Now, let's set up a conversation for our bot:

```python
messages = [
  {"role": "system", "content": "You are a bot that responds to weather queries. You should reply with the unit used in the queried location."},
  {"role": "user", "content": "Hey, what's the temperature in Paris right now?"}
]
```

Now, let's apply the chat template and generate a response:

```python
inputs = tokenizer.apply_chat_template(messages, chat_template="tool_use", tools=tools, add_generation_prompt=True, return_dict=True, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0][len(inputs["input_ids"][0]):]))
```

And we get:

```text
<tool_call>
{"arguments": {"location": "Paris, France", "unit": "celsius"}, "name": "get_current_temperature"}
</tool_call><|im_end|>
```

The model has called the function with valid arguments, in the format requested by the function docstring. It has
inferred that we're most likely referring to the Paris in France, and it remembered that, as the home of SI units,
the temperature in France should certainly be displayed in Celsius.

Let's append the model's tool call to the conversation. Note that we generate a random `tool_call_id` here.


key: _node_content
string_value:
{"id_": "b0a18978-a762-4bb0-baaf-517897c124f7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ec746d55-c0f6-419e-b8c9-b174ad8c087c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "38b7fcdf7c28ca895a9de8fff8d6bba2b5b1ca6f8e3ebcae859ab9ef2c9a645f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "46239ce9-0666-45f3-9e39-f6310134ac82", "node_type": "1", "metadata": {}, "hash": "b34d5bffbabd22b89741ac038fdc025dbbb8b2ac212943994a3efcce839b85a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14811, "end_char_idx": 17314, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current wind speed at the given location in km/h, as a float.
    """
    return 6.  # A real function should probably actually get the wind speed!

tools = [get_current_temperature, get_current_wind_speed]
```

Now, let's set up a conversation for our bot:

```python
messages = [
  {"role": "system", "content": "You are a bot that responds to weather queries. You should reply with the unit used in the queried location."},
  {"role": "user", "content": "Hey, what's the temperature in Paris right now?"}
]
```

Now, let's apply the chat template and generate a response:

```python
inputs = tokenizer.apply_chat_template(messages, chat_template="tool_use", tools=tools, add_generation_prompt=True, return_dict=True, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0][len(inputs["input_ids"][0]):]))
```

And we get:

```text
<tool_call>
{"arguments": {"location": "Paris, France", "unit": "celsius"}, "name": "get_current_temperature"}
</tool_call><|im_end|>
```

The model has called the function with valid arguments, in the format requested by the function docstring. It has
inferred that we're most likely referring to the Paris in France, and it remembered that, as the home of SI units,
the temperature in France should certainly be displayed in Celsius.

Let's append the model's tool call to the conversation. Note that we generate a random `tool_call_id` here. These IDs
are not used by all models, but they allow models to issue multiple tool calls at once and keep track of which response
corresponds to which call. You can generate them any way you like, but they should be unique within each chat.

```python
tool_call_id = "vAHdf3"  # Random ID, should be unique for each tool call
tool_call = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
messages.append({"role": "assistant", "tool_calls": [{"id": tool_call_id, "type": "function", "function": tool_call}]})
```


Now that we've added the tool call to the conversation, we can call the function and append the result to the
conversation. Since we're just using a dummy function for this example that always returns 22.0, we can just append 
that result directly. Again, note the `tool_call_id` - this should match the ID used in the tool call above.


key: _node_content
string_value:
{"id_": "46239ce9-0666-45f3-9e39-f6310134ac82", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b0a18978-a762-4bb0-baaf-517897c124f7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "04906b7739c50d99bd1f45ff58d86746eb06229317587e5d5c89c40d96a039be", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4180a3b7-ffe3-4ccd-bdf2-d08af12ed4f1", "node_type": "1", "metadata": {}, "hash": "5260ffdc090236272bcf58e7e83915169d4a9b462596dcbcb7f9733747a64dc5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16117, "end_char_idx": 19434, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It has
inferred that we're most likely referring to the Paris in France, and it remembered that, as the home of SI units,
the temperature in France should certainly be displayed in Celsius.

Let's append the model's tool call to the conversation. Note that we generate a random `tool_call_id` here. These IDs
are not used by all models, but they allow models to issue multiple tool calls at once and keep track of which response
corresponds to which call. You can generate them any way you like, but they should be unique within each chat.

```python
tool_call_id = "vAHdf3"  # Random ID, should be unique for each tool call
tool_call = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
messages.append({"role": "assistant", "tool_calls": [{"id": tool_call_id, "type": "function", "function": tool_call}]})
```


Now that we've added the tool call to the conversation, we can call the function and append the result to the
conversation. Since we're just using a dummy function for this example that always returns 22.0, we can just append 
that result directly. Again, note the `tool_call_id` - this should match the ID used in the tool call above.

```python
messages.append({"role": "tool", "tool_call_id": tool_call_id, "name": "get_current_temperature", "content": "22.0"})
```

Finally, let's let the assistant read the function outputs and continue chatting with the user:

```python
inputs = tokenizer.apply_chat_template(messages, chat_template="tool_use", tools=tools, add_generation_prompt=True, return_dict=True, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0][len(inputs["input_ids"][0]):]))
```

And we get:

```text
The current temperature in Paris, France is 22.0 ° Celsius.<|im_end|>
```

Although this was a simple demo with dummy tools and a single call, the same technique works with 
multiple real tools and longer conversations. This can be a powerful way to extend the capabilities of conversational
agents with real-time information, computational tools like calculators, or access to large databases.

<Tip>
Not all of the tool-calling features shown above are used by all models. Some use tool call IDs, others simply use the function name and
match tool calls to results using the ordering, and there are several models that use neither and only issue one tool 
call at a time to avoid confusion. If you want your code to be compatible across as many models as possible, we 
recommend structuring your tools calls like we've shown here, and returning tool results in the order that
they were issued by the model. The chat templates on each model should handle the rest.
</Tip>

### Understanding tool schemas

Each function you pass to the `tools` argument of `apply_chat_template` is converted into a 
[JSON schema](https://json-schema.org/learn/getting-started-step-by-step). These schemas
are then passed to the model chat template. In other words, tool-use models do not see your functions directly, and they
never see the actual code inside them. What they care about is the function **definitions** and the **arguments** they
need to pass to them - they care about what the tools do and how to use them, not how they work!


key: _node_content
string_value:
{"id_": "4180a3b7-ffe3-4ccd-bdf2-d08af12ed4f1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "46239ce9-0666-45f3-9e39-f6310134ac82", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "ebb68cf5ecf88abb4e22c20b2a7aedd95cd4f5a04c82718e05791c6b7d8e8784", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "24d9ed0e-5ffc-4f93-93c2-8162a8b7d833", "node_type": "1", "metadata": {}, "hash": "04b122433c2f617b76cf3cc481e1e23fefc0e56c46cf8ad4fddc04dfb3e46efd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18130, "end_char_idx": 21582, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This can be a powerful way to extend the capabilities of conversational
agents with real-time information, computational tools like calculators, or access to large databases.

<Tip>
Not all of the tool-calling features shown above are used by all models. Some use tool call IDs, others simply use the function name and
match tool calls to results using the ordering, and there are several models that use neither and only issue one tool 
call at a time to avoid confusion. If you want your code to be compatible across as many models as possible, we 
recommend structuring your tools calls like we've shown here, and returning tool results in the order that
they were issued by the model. The chat templates on each model should handle the rest.
</Tip>

### Understanding tool schemas

Each function you pass to the `tools` argument of `apply_chat_template` is converted into a 
[JSON schema](https://json-schema.org/learn/getting-started-step-by-step). These schemas
are then passed to the model chat template. In other words, tool-use models do not see your functions directly, and they
never see the actual code inside them. What they care about is the function **definitions** and the **arguments** they
need to pass to them - they care about what the tools do and how to use them, not how they work! It is up to you
to read their outputs, detect if they have requested to use a tool, pass their arguments to the tool function, and
return the response in the chat.

Generating JSON schemas to pass to the template should be automatic and invisible as long as your functions
follow the specification above, but if you encounter problems, or you simply want more control over the conversion, 
you can handle the conversion manually. Here is an example of a manual schema conversion.

```python
from transformers.utils import get_json_schema

def multiply(a: float, b: float):
    """
    A function that multiplies two numbers
    
    Args:
        a: The first number to multiply
        b: The second number to multiply
    """
    return a * b

schema = get_json_schema(multiply)
print(schema)
```

This will yield:

```json
{
  "type": "function", 
  "function": {
    "name": "multiply", 
    "description": "A function that multiplies two numbers", 
    "parameters": {
      "type": "object", 
      "properties": {
        "a": {
          "type": "number", 
          "description": "The first number to multiply"
        }, 
        "b": {
          "type": "number",
          "description": "The second number to multiply"
        }
      }, 
      "required": ["a", "b"]
    }
  }
}
```

If you wish, you can edit these schemas, or even write them from scratch yourself without using `get_json_schema` at 
all. JSON schemas can be passed directly to the `tools` argument of 
`apply_chat_template` - this gives you a lot of power to define precise schemas for more complex functions. Be careful,
though - the more complex your schemas, the more likely the model is to get confused when dealing with them! We 
recommend simple function signatures where possible, keeping arguments (and especially complex, nested arguments) 
to a minimum.

Here is an example of defining schemas by hand, and passing them directly to `apply_chat_template`:

```python
# A simple function that takes no arguments
current_time = {
  "type": "function", 
  "function": {
    "name": "current_time",
    "description": "Get the current local time as a string.


key: _node_content
string_value:
{"id_": "24d9ed0e-5ffc-4f93-93c2-8162a8b7d833", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4180a3b7-ffe3-4ccd-bdf2-d08af12ed4f1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8d69a388d2a3eb2001035dd623088e42d7d913fe70c0180c95d3800083eaff42", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "04002efc-e1db-4a20-a7df-93ac132f0be2", "node_type": "1", "metadata": {}, "hash": "4cca5d89125ec3c144f66865f56ebd168b4054814bb4ea2ce9f279916e52aedc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19916, "end_char_idx": 23079, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers.utils import get_json_schema

def multiply(a: float, b: float):
    """
    A function that multiplies two numbers
    
    Args:
        a: The first number to multiply
        b: The second number to multiply
    """
    return a * b

schema = get_json_schema(multiply)
print(schema)
```

This will yield:

```json
{
  "type": "function", 
  "function": {
    "name": "multiply", 
    "description": "A function that multiplies two numbers", 
    "parameters": {
      "type": "object", 
      "properties": {
        "a": {
          "type": "number", 
          "description": "The first number to multiply"
        }, 
        "b": {
          "type": "number",
          "description": "The second number to multiply"
        }
      }, 
      "required": ["a", "b"]
    }
  }
}
```

If you wish, you can edit these schemas, or even write them from scratch yourself without using `get_json_schema` at 
all. JSON schemas can be passed directly to the `tools` argument of 
`apply_chat_template` - this gives you a lot of power to define precise schemas for more complex functions. Be careful,
though - the more complex your schemas, the more likely the model is to get confused when dealing with them! We 
recommend simple function signatures where possible, keeping arguments (and especially complex, nested arguments) 
to a minimum.

Here is an example of defining schemas by hand, and passing them directly to `apply_chat_template`:

```python
# A simple function that takes no arguments
current_time = {
  "type": "function", 
  "function": {
    "name": "current_time",
    "description": "Get the current local time as a string.",
    "parameters": {
      'type': 'object',
      'properties': {}
    }
  }
}

# A more complete function that takes two numerical arguments
multiply = {
  'type': 'function',
  'function': {
    'name': 'multiply',
    'description': 'A function that multiplies two numbers', 
    'parameters': {
      'type': 'object', 
      'properties': {
        'a': {
          'type': 'number',
          'description': 'The first number to multiply'
        }, 
        'b': {
          'type': 'number', 'description': 'The second number to multiply'
        }
      }, 
      'required': ['a', 'b']
    }
  }
}

model_input = tokenizer.apply_chat_template(
    messages,
    tools = [current_time, multiply]
)
```

## Advanced: Retrieval-augmented generation

"Retrieval-augmented generation" or "RAG" LLMs can search a corpus of documents for information before responding
to a query. This allows models to vastly expand their knowledge base beyond their limited context size. Our 
recommendation for RAG models is that their template
should accept a `documents` argument. This should be a list of documents, where each "document"
is a single dict with `title` and `contents` keys, both of which are strings. Because this format is much simpler
than the JSON schemas used for tools, no helper functions are necessary.

Here's an example of a RAG template in action:

```python
document1 = {
    "title": "The Moon: Our Age-Old Foe",
    "contents": "Man has always dreamed of destroying the moon.


key: _node_content
string_value:
{"id_": "04002efc-e1db-4a20-a7df-93ac132f0be2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "24d9ed0e-5ffc-4f93-93c2-8162a8b7d833", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "42470a7ddbcf2e1b6bd05add1770ea8e63514d4f424a04e3f83509680643d71c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e5d5497-fb87-44f8-8fae-a3ee4b67526c", "node_type": "1", "metadata": {}, "hash": "b9957649c918f82549f1288887196479431035b7b872f14ae4124ec8b731dc7c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21582, "end_char_idx": 24637, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
    "parameters": {
      'type': 'object',
      'properties': {}
    }
  }
}

# A more complete function that takes two numerical arguments
multiply = {
  'type': 'function',
  'function': {
    'name': 'multiply',
    'description': 'A function that multiplies two numbers', 
    'parameters': {
      'type': 'object', 
      'properties': {
        'a': {
          'type': 'number',
          'description': 'The first number to multiply'
        }, 
        'b': {
          'type': 'number', 'description': 'The second number to multiply'
        }
      }, 
      'required': ['a', 'b']
    }
  }
}

model_input = tokenizer.apply_chat_template(
    messages,
    tools = [current_time, multiply]
)
```

## Advanced: Retrieval-augmented generation

"Retrieval-augmented generation" or "RAG" LLMs can search a corpus of documents for information before responding
to a query. This allows models to vastly expand their knowledge base beyond their limited context size. Our 
recommendation for RAG models is that their template
should accept a `documents` argument. This should be a list of documents, where each "document"
is a single dict with `title` and `contents` keys, both of which are strings. Because this format is much simpler
than the JSON schemas used for tools, no helper functions are necessary.

Here's an example of a RAG template in action:

```python
document1 = {
    "title": "The Moon: Our Age-Old Foe",
    "contents": "Man has always dreamed of destroying the moon. In this essay, I shall..."
}

document2 = {
    "title": "The Sun: Our Age-Old Friend",
    "contents": "Although often underappreciated, the sun provides several notable benefits..."
}

model_input = tokenizer.apply_chat_template(
    messages,
    documents=[document1, document2]
)
```

## Advanced: How do chat templates work?

The chat template for a model is stored on the `tokenizer.chat_template` attribute. If no chat template is set, the
default template for that model class is used instead. Let's take a look at the template for `BlenderBot`:

```python

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")

>>> tokenizer.chat_template
"{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ '  ' }}{% endif %}{% endfor %}{{ eos_token }}"
```

That's kind of intimidating. Let's clean it up a little to make it more readable. In the process, though, we also make
sure that the newlines and indentation we add don't end up being included in the template output - see the tip on
[trimming whitespace](#trimming-whitespace) below!

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- ' ' }}
    {%- endif %}
    {{- message['content'] }}
    {%- if not loop.last %}
        {{- '  ' }}
    {%- endif %}
{%- endfor %}
{{- eos_token }}
```

If you've never seen one of these before, this is a [Jinja template](https://jinja.palletsprojects.com/en/3.1.x/templates/).


key: _node_content
string_value:
{"id_": "0e5d5497-fb87-44f8-8fae-a3ee4b67526c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "04002efc-e1db-4a20-a7df-93ac132f0be2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "86567b7f34060d5dc025df9fce5e6764a92f3bf962da161bc573753a503fb2d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "270d6860-0401-4bd9-9fca-f9ce13927614", "node_type": "1", "metadata": {}, "hash": "758f1b8e3dcb8fc9afe129f6507242ba3ccb97b08e8454439838e7227e8aae66", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23412, "end_char_idx": 25957, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The chat template for a model is stored on the `tokenizer.chat_template` attribute. If no chat template is set, the
default template for that model class is used instead. Let's take a look at the template for `BlenderBot`:

```python

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")

>>> tokenizer.chat_template
"{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ '  ' }}{% endif %}{% endfor %}{{ eos_token }}"
```

That's kind of intimidating. Let's clean it up a little to make it more readable. In the process, though, we also make
sure that the newlines and indentation we add don't end up being included in the template output - see the tip on
[trimming whitespace](#trimming-whitespace) below!

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- ' ' }}
    {%- endif %}
    {{- message['content'] }}
    {%- if not loop.last %}
        {{- '  ' }}
    {%- endif %}
{%- endfor %}
{{- eos_token }}
```

If you've never seen one of these before, this is a [Jinja template](https://jinja.palletsprojects.com/en/3.1.x/templates/).
Jinja is a templating language that allows you to write simple code that generates text. In many ways, the code and
syntax resembles Python. In pure Python, this template would look something like this:

```python
for idx, message in enumerate(messages):
    if message['role'] == 'user':
        print(' ')
    print(message['content'])
    if not idx == len(messages) - 1:  # Check for the last message in the conversation
        print('  ')
print(eos_token)
```

Effectively, the template does three things:
1. For each message, if the message is a user message, add a blank space before it, otherwise print nothing.
2. Add the message content
3. If the message is not the last message, add two spaces after it. After the final message, print the EOS token.

This is a pretty simple template - it doesn't add any control tokens, and it doesn't support "system" messages, which 
are a common way to give the model directives about how it should behave in the subsequent conversation.
But Jinja gives you a lot of flexibility to do those things! Let's see a Jinja template that can format inputs
similarly to the way LLaMA formats them (note that the real LLaMA template includes handling for default system
messages and slightly different system message handling in general - don't use this one in your actual code!)


key: _node_content
string_value:
{"id_": "270d6860-0401-4bd9-9fca-f9ce13927614", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e5d5497-fb87-44f8-8fae-a3ee4b67526c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "bb536d209752276417c458a7b387306673ff94d0aa72e208ae142e0e2cf62f52", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ab0421c-bfae-413a-8ff2-658bda3ad5c0", "node_type": "1", "metadata": {}, "hash": "270533b1c4006fc61fde43555ea9cbabb15a108a44d25cae80851f304a81defd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24638, "end_char_idx": 27623, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Jinja is a templating language that allows you to write simple code that generates text. In many ways, the code and
syntax resembles Python. In pure Python, this template would look something like this:

```python
for idx, message in enumerate(messages):
    if message['role'] == 'user':
        print(' ')
    print(message['content'])
    if not idx == len(messages) - 1:  # Check for the last message in the conversation
        print('  ')
print(eos_token)
```

Effectively, the template does three things:
1. For each message, if the message is a user message, add a blank space before it, otherwise print nothing.
2. Add the message content
3. If the message is not the last message, add two spaces after it. After the final message, print the EOS token.

This is a pretty simple template - it doesn't add any control tokens, and it doesn't support "system" messages, which 
are a common way to give the model directives about how it should behave in the subsequent conversation.
But Jinja gives you a lot of flexibility to do those things! Let's see a Jinja template that can format inputs
similarly to the way LLaMA formats them (note that the real LLaMA template includes handling for default system
messages and slightly different system message handling in general - don't use this one in your actual code!)

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- bos_token + '[INST] ' + message['content'] + ' [/INST]' }}
    {%- elif message['role'] == 'system' %}
        {{- '<<SYS>>\\n' + message['content'] + '\\n<</SYS>>\\n\\n' }}
    {%- elif message['role'] == 'assistant' %}
        {{- ' '  + message['content'] + ' ' + eos_token }}
    {%- endif %}
{%- endfor %}
```

Hopefully if you stare at this for a little bit you can see what this template is doing - it adds specific tokens based
on the "role" of each message, which represents who sent it. User, assistant and system messages are clearly
distinguishable to the model because of the tokens they're wrapped in.

## Advanced: Adding and editing chat templates

### How do I create a chat template?

Simple, just write a jinja template and set `tokenizer.chat_template`. You may find it easier to start with an 
existing template from another model and simply edit it for your needs! For example, we could take the LLaMA template
above and add "[ASST]" and "[/ASST]" to assistant messages:

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- bos_token + '[INST] ' + message['content'].strip() + ' [/INST]' }}
    {%- elif message['role'] == 'system' %}
        {{- '<<SYS>>\\n' + message['content'].strip() + '\\n<</SYS>>\\n\\n' }}
    {%- elif message['role'] == 'assistant' %}
        {{- '[ASST] '  + message['content'] + ' [/ASST]' + eos_token }}
    {%- endif %}
{%- endfor %}
```

Now, simply set the `tokenizer.chat_template` attribute. Next time you use [`~PreTrainedTokenizer.apply_chat_template`], it will
use your new template!


key: _node_content
string_value:
{"id_": "1ab0421c-bfae-413a-8ff2-658bda3ad5c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "270d6860-0401-4bd9-9fca-f9ce13927614", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "459e7a4e51a5532b4593e55be584025d65edd1f7dd1ea77c03dadb8805d0ed63", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b67c8ce1-49f3-4640-acf3-5d34aee2c364", "node_type": "1", "metadata": {}, "hash": "043e4963ac31fab00c5ce61db3f70b730089dbe3815a0f4332c0450a859f16e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26543, "end_char_idx": 29910, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
User, assistant and system messages are clearly
distinguishable to the model because of the tokens they're wrapped in.

## Advanced: Adding and editing chat templates

### How do I create a chat template?

Simple, just write a jinja template and set `tokenizer.chat_template`. You may find it easier to start with an 
existing template from another model and simply edit it for your needs! For example, we could take the LLaMA template
above and add "[ASST]" and "[/ASST]" to assistant messages:

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- bos_token + '[INST] ' + message['content'].strip() + ' [/INST]' }}
    {%- elif message['role'] == 'system' %}
        {{- '<<SYS>>\\n' + message['content'].strip() + '\\n<</SYS>>\\n\\n' }}
    {%- elif message['role'] == 'assistant' %}
        {{- '[ASST] '  + message['content'] + ' [/ASST]' + eos_token }}
    {%- endif %}
{%- endfor %}
```

Now, simply set the `tokenizer.chat_template` attribute. Next time you use [`~PreTrainedTokenizer.apply_chat_template`], it will
use your new template! This attribute will be saved in the `tokenizer_config.json` file, so you can use
[`~utils.PushToHubMixin.push_to_hub`] to upload your new template to the Hub and make sure everyone's using the right
template for your model!

```python
template = tokenizer.chat_template
template = template.replace("SYS", "SYSTEM")  # Change the system token
tokenizer.chat_template = template  # Set the new template
tokenizer.push_to_hub("model_name")  # Upload your new template to the Hub!
```

The method [`~PreTrainedTokenizer.apply_chat_template`] which uses your chat template is called by the [`TextGenerationPipeline`] class, so 
once you set the correct chat template, your model will automatically become compatible with [`TextGenerationPipeline`].

<Tip>
If you're fine-tuning a model for chat, in addition to setting a chat template, you should probably add any new chat
control tokens as special tokens in the tokenizer. Special tokens are never split, 
ensuring that your control tokens are always handled as single tokens rather than being tokenized in pieces. You 
should also set the tokenizer's `eos_token` attribute to the token that marks the end of assistant generations in your
template. This will ensure that text generation tools can correctly figure out when to stop generating text.
</Tip>


### Why do some models have multiple templates?

Some models use different templates for different use cases. For example, they might use one template for normal chat
and another for tool-use, or retrieval-augmented generation. In these cases, `tokenizer.chat_template` is a dictionary.
This can cause some confusion, and where possible, we recommend using a single template for all use-cases. You can use
Jinja statements like `if tools is defined` and `{% macro %}` definitions to easily wrap multiple code paths in a
single template.

When a tokenizer has multiple templates, `tokenizer.chat_template` will be a `dict`, where each key is the name
of a template. The `apply_chat_template` method has special handling for certain template names: Specifically, it will
look for a template named `default` in most cases, and will raise an error if it can't find one. However, if a template
named `tool_use` exists when the user has passed a `tools` argument, it will use that instead.


key: _node_content
string_value:
{"id_": "b67c8ce1-49f3-4640-acf3-5d34aee2c364", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1ab0421c-bfae-413a-8ff2-658bda3ad5c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "c27a414fd6bcccbe5a5159fb6e9fcf35cb2b68994b6bc836941b60cd9d27cde9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c0ef51a0-800f-4846-b5be-0848993154b7", "node_type": "1", "metadata": {}, "hash": "59985f350bc611905b37d72e60b0e15cdc0e7e24a242f61ec8c5bba498e44a4a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28101, "end_char_idx": 31557, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

The method [`~PreTrainedTokenizer.apply_chat_template`] which uses your chat template is called by the [`TextGenerationPipeline`] class, so 
once you set the correct chat template, your model will automatically become compatible with [`TextGenerationPipeline`].

<Tip>
If you're fine-tuning a model for chat, in addition to setting a chat template, you should probably add any new chat
control tokens as special tokens in the tokenizer. Special tokens are never split, 
ensuring that your control tokens are always handled as single tokens rather than being tokenized in pieces. You 
should also set the tokenizer's `eos_token` attribute to the token that marks the end of assistant generations in your
template. This will ensure that text generation tools can correctly figure out when to stop generating text.
</Tip>


### Why do some models have multiple templates?

Some models use different templates for different use cases. For example, they might use one template for normal chat
and another for tool-use, or retrieval-augmented generation. In these cases, `tokenizer.chat_template` is a dictionary.
This can cause some confusion, and where possible, we recommend using a single template for all use-cases. You can use
Jinja statements like `if tools is defined` and `{% macro %}` definitions to easily wrap multiple code paths in a
single template.

When a tokenizer has multiple templates, `tokenizer.chat_template` will be a `dict`, where each key is the name
of a template. The `apply_chat_template` method has special handling for certain template names: Specifically, it will
look for a template named `default` in most cases, and will raise an error if it can't find one. However, if a template
named `tool_use` exists when the user has passed a `tools` argument, it will use that instead. To access templates
with other names, pass the name of the template you want to the `chat_template` argument of
`apply_chat_template()`.

We find that this can be a bit confusing for users, though - so if you're writing a template yourself, we recommend
trying to put it all in a single template where possible!

### What template should I use?

When setting the template for a model that's already been trained for chat, you should ensure that the template
exactly matches the message formatting that the model saw during training, or else you will probably experience
performance degradation. This is true even if you're training the model further - you will probably get the best 
performance if you keep the chat tokens constant. This is very analogous to tokenization - you generally get the
best performance for inference or fine-tuning when you precisely match the tokenization used during training.

If you're training a model from scratch, or fine-tuning a base language model for chat, on the other hand,
you have a lot of freedom to choose an appropriate template! LLMs are smart enough to learn to handle lots of different
input formats. One popular choice is the `ChatML` format, and this is a good, flexible choice for many use-cases. 
It looks like this:

```
{%- for message in messages %}
    {{- '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n' }}
{%- endfor %}
```

If you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes
handy support for [generation prompts](#what-are-generation-prompts), but note that it doesn't add BOS or EOS tokens!


key: _node_content
string_value:
{"id_": "c0ef51a0-800f-4846-b5be-0848993154b7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b67c8ce1-49f3-4640-acf3-5d34aee2c364", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "a26fe0adbf9a4ba284d1cb7afb550dcd7e9439d963fb7c5324c0982d083adb59", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5872dda-31ea-4617-83eb-190e22a3121a", "node_type": "1", "metadata": {}, "hash": "caa34dc4a20f5c85f2bec8c6d87a48d24083b0f354900c893c3fc57722c003e6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29911, "end_char_idx": 33053, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To access templates
with other names, pass the name of the template you want to the `chat_template` argument of
`apply_chat_template()`.

We find that this can be a bit confusing for users, though - so if you're writing a template yourself, we recommend
trying to put it all in a single template where possible!

### What template should I use?

When setting the template for a model that's already been trained for chat, you should ensure that the template
exactly matches the message formatting that the model saw during training, or else you will probably experience
performance degradation. This is true even if you're training the model further - you will probably get the best 
performance if you keep the chat tokens constant. This is very analogous to tokenization - you generally get the
best performance for inference or fine-tuning when you precisely match the tokenization used during training.

If you're training a model from scratch, or fine-tuning a base language model for chat, on the other hand,
you have a lot of freedom to choose an appropriate template! LLMs are smart enough to learn to handle lots of different
input formats. One popular choice is the `ChatML` format, and this is a good, flexible choice for many use-cases. 
It looks like this:

```
{%- for message in messages %}
    {{- '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n' }}
{%- endfor %}
```

If you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes
handy support for [generation prompts](#what-are-generation-prompts), but note that it doesn't add BOS or EOS tokens!
If your model expects those, they won't be added automatically by `apply_chat_template` - in other words, the
text will be tokenized with `add_special_tokens=False`. This is to avoid potential conflicts between the template and
the `add_special_tokens` logic. If your model expects special tokens, make sure to add them to the template!

```python
tokenizer.chat_template = "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
```

This template wraps each message in `<|im_start|>` and `<|im_end|>` tokens, and simply writes the role as a string, which
allows for flexibility in the roles you train with. The output looks like this:

```text
<|im_start|>system
You are a helpful chatbot that will do its best not to say anything so stupid that people tweet about it.<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
I'm doing great!<|im_end|>
```

The "user", "system" and "assistant" roles are the standard for chat, and we recommend using them when it makes sense,
particularly if you want your model to operate well with [`TextGenerationPipeline`]. However, you are not limited
to these roles - templating is extremely flexible, and any string can be a role.

### I want to add some chat templates! How should I get started?


key: _node_content
string_value:
{"id_": "c5872dda-31ea-4617-83eb-190e22a3121a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c0ef51a0-800f-4846-b5be-0848993154b7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "762750e5441b0d976cb1c1787948d6b6eaa610ddb720ec71bd18f869886f8b40", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ce0146dc-854c-4098-ad92-f07a121aca5b", "node_type": "1", "metadata": {}, "hash": "a4484dbb6246d68a70b63b9bc7fb67485edf7831dddc0ffe1589ef0594075406", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31558, "end_char_idx": 34824, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If your model expects those, they won't be added automatically by `apply_chat_template` - in other words, the
text will be tokenized with `add_special_tokens=False`. This is to avoid potential conflicts between the template and
the `add_special_tokens` logic. If your model expects special tokens, make sure to add them to the template!

```python
tokenizer.chat_template = "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
```

This template wraps each message in `<|im_start|>` and `<|im_end|>` tokens, and simply writes the role as a string, which
allows for flexibility in the roles you train with. The output looks like this:

```text
<|im_start|>system
You are a helpful chatbot that will do its best not to say anything so stupid that people tweet about it.<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
I'm doing great!<|im_end|>
```

The "user", "system" and "assistant" roles are the standard for chat, and we recommend using them when it makes sense,
particularly if you want your model to operate well with [`TextGenerationPipeline`]. However, you are not limited
to these roles - templating is extremely flexible, and any string can be a role.

### I want to add some chat templates! How should I get started?

If you have any chat models, you should set their `tokenizer.chat_template` attribute and test it using
[`~PreTrainedTokenizer.apply_chat_template`], then push the updated tokenizer to the Hub. This applies even if you're
not the model owner - if you're using a model with an empty chat template, or one that's still using the default class
template, please open a [pull request](https://huggingface.co/docs/hub/repositories-pull-requests-discussions) to the model repository so that this attribute can be set properly!

Once the attribute is set, that's it, you're done! `tokenizer.apply_chat_template` will now work correctly for that
model, which means it is also automatically supported in places like `TextGenerationPipeline`!

By ensuring that models have this attribute, we can make sure that the whole community gets to use the full power of
open-source models. Formatting mismatches have been haunting the field and silently harming performance for too long - 
it's time to put an end to them!

## Advanced: Template writing tips

If you're unfamiliar with Jinja, we generally find that the easiest way to write a chat template is to first
write a short Python script that formats messages the way you want, and then convert that script into a template.

Remember that the template handler will receive the conversation history as a variable called `messages`.  
You will be able to access `messages` in your template just like you can in Python, which means you can loop over 
it with `{% for message in messages %}` or access individual messages with `{{ messages[0] }}`, for example.

You can also use the following tips to convert your code to Jinja:

### Trimming whitespace

By default, Jinja will print any whitespace that comes before or after a block.


key: _node_content
string_value:
{"id_": "ce0146dc-854c-4098-ad92-f07a121aca5b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5872dda-31ea-4617-83eb-190e22a3121a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "089f19319148e2d0700edc7a45e3651011cb6b9a32942213f7678f598e3de312", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "77e2291e-d56f-45ff-ac08-414d284f4fde", "node_type": "1", "metadata": {}, "hash": "a1f056b7fc590e7523e5eee2012e788efb33d72eaeee36e8f7bda138ec72cdbc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32989, "end_char_idx": 36341, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### I want to add some chat templates! How should I get started?

If you have any chat models, you should set their `tokenizer.chat_template` attribute and test it using
[`~PreTrainedTokenizer.apply_chat_template`], then push the updated tokenizer to the Hub. This applies even if you're
not the model owner - if you're using a model with an empty chat template, or one that's still using the default class
template, please open a [pull request](https://huggingface.co/docs/hub/repositories-pull-requests-discussions) to the model repository so that this attribute can be set properly!

Once the attribute is set, that's it, you're done! `tokenizer.apply_chat_template` will now work correctly for that
model, which means it is also automatically supported in places like `TextGenerationPipeline`!

By ensuring that models have this attribute, we can make sure that the whole community gets to use the full power of
open-source models. Formatting mismatches have been haunting the field and silently harming performance for too long - 
it's time to put an end to them!

## Advanced: Template writing tips

If you're unfamiliar with Jinja, we generally find that the easiest way to write a chat template is to first
write a short Python script that formats messages the way you want, and then convert that script into a template.

Remember that the template handler will receive the conversation history as a variable called `messages`.  
You will be able to access `messages` in your template just like you can in Python, which means you can loop over 
it with `{% for message in messages %}` or access individual messages with `{{ messages[0] }}`, for example.

You can also use the following tips to convert your code to Jinja:

### Trimming whitespace

By default, Jinja will print any whitespace that comes before or after a block. This can be a problem for chat
templates, which generally want to be very precise with whitespace! To avoid this, we strongly recommend writing
your templates like this:

```
{%- for message in messages %}
    {{- message['role'] + message['content'] }}
{%- endfor %}
```

rather than like this:

```
{% for message in messages %}
    {{ message['role'] + message['content'] }}
{% endfor %}
```

Adding `-` will strip any whitespace that comes before the block. The second example looks innocent, but the newline
and indentation may end up being included in the output, which is probably not what you want!

### For loops

For loops in Jinja look like this:

```
{%- for message in messages %}
    {{- message['content'] }}
{%- endfor %}
```

Note that whatever's inside the {{ expression block }} will be printed to the output. You can use operators like
`+` to combine strings inside expression blocks.

### If statements

If statements in Jinja look like this:

```
{%- if message['role'] == 'user' %}
    {{- message['content'] }}
{%- endif %}
```

Note how where Python uses whitespace to mark the beginnings and ends of `for` and `if` blocks, Jinja requires you
to explicitly end them with `{% endfor %}` and `{% endif %}`.

### Special variables

Inside your template, you will have access to the list of `messages`, but you can also access several other special
variables. These include special tokens like `bos_token` and `eos_token`, as well as the `add_generation_prompt`
variable that we discussed above.


key: _node_content
string_value:
{"id_": "77e2291e-d56f-45ff-ac08-414d284f4fde", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ce0146dc-854c-4098-ad92-f07a121aca5b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "6e70a068347b2dff569c2b5f85186c276b6c6e2725defc8ee131aba97db12480", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b185b548-ac1a-4679-861e-39269998cec0", "node_type": "1", "metadata": {}, "hash": "53cecd8a1f18eaf777d7b45aa6e9c175cf17d232a3a2e8a48e85ac563cb9a228", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34652, "end_char_idx": 37941, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can also use the following tips to convert your code to Jinja:

### Trimming whitespace

By default, Jinja will print any whitespace that comes before or after a block. This can be a problem for chat
templates, which generally want to be very precise with whitespace! To avoid this, we strongly recommend writing
your templates like this:

```
{%- for message in messages %}
    {{- message['role'] + message['content'] }}
{%- endfor %}
```

rather than like this:

```
{% for message in messages %}
    {{ message['role'] + message['content'] }}
{% endfor %}
```

Adding `-` will strip any whitespace that comes before the block. The second example looks innocent, but the newline
and indentation may end up being included in the output, which is probably not what you want!

### For loops

For loops in Jinja look like this:

```
{%- for message in messages %}
    {{- message['content'] }}
{%- endfor %}
```

Note that whatever's inside the {{ expression block }} will be printed to the output. You can use operators like
`+` to combine strings inside expression blocks.

### If statements

If statements in Jinja look like this:

```
{%- if message['role'] == 'user' %}
    {{- message['content'] }}
{%- endif %}
```

Note how where Python uses whitespace to mark the beginnings and ends of `for` and `if` blocks, Jinja requires you
to explicitly end them with `{% endfor %}` and `{% endif %}`.

### Special variables

Inside your template, you will have access to the list of `messages`, but you can also access several other special
variables. These include special tokens like `bos_token` and `eos_token`, as well as the `add_generation_prompt`
variable that we discussed above. You can also use the `loop` variable to access information about the current loop
iteration, for example  using `{% if loop.last %}` to check if the current message is the last message in the 
conversation. Here's an example that puts these ideas together to add a generation prompt at the end of the
conversation if add_generation_prompt is `True`:

```
{%- if loop.last and add_generation_prompt %}
    {{- bos_token + 'Assistant:\n' }}
{%- endif %}
```

### Compatibility with non-Python Jinja

There are multiple implementations of Jinja in various languages. They generally have the same syntax,
but a key difference is that when you're writing a template in Python you can use Python methods, such as
`.lower()` on strings or `.items()` on dicts. This will break if someone tries to use your template on a non-Python
implementation of Jinja. Non-Python implementations are particularly common in deployment environments, where JS
and Rust are very popular. 

Don't panic, though! There are a few easy changes you can make to your templates to ensure they're compatible across
all implementations of Jinja:

- Replace Python methods with Jinja filters. These usually have the same name, for example `string.lower()` becomes
  `string|lower`, and `dict.items()` becomes `dict|items`. One notable change is that `string.strip()` becomes `string|trim`.
  See the [list of built-in filters](https://jinja.palletsprojects.com/en/3.1.x/templates/#builtin-filters)
  in the Jinja documentation for more.
- Replace `True`, `False` and `None`, which are Python-specific, with `true`, `false` and `none`.


key: _node_content
string_value:
{"id_": "b185b548-ac1a-4679-861e-39269998cec0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "77e2291e-d56f-45ff-ac08-414d284f4fde", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers"}, "hash": "6e79d51c615d7821d1cda5cdf3e3f93723bc5b87f0913e82d26cce4f06d01a8b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36342, "end_char_idx": 38175, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can also use the `loop` variable to access information about the current loop
iteration, for example  using `{% if loop.last %}` to check if the current message is the last message in the 
conversation. Here's an example that puts these ideas together to add a generation prompt at the end of the
conversation if add_generation_prompt is `True`:

```
{%- if loop.last and add_generation_prompt %}
    {{- bos_token + 'Assistant:\n' }}
{%- endif %}
```

### Compatibility with non-Python Jinja

There are multiple implementations of Jinja in various languages. They generally have the same syntax,
but a key difference is that when you're writing a template in Python you can use Python methods, such as
`.lower()` on strings or `.items()` on dicts. This will break if someone tries to use your template on a non-Python
implementation of Jinja. Non-Python implementations are particularly common in deployment environments, where JS
and Rust are very popular. 

Don't panic, though! There are a few easy changes you can make to your templates to ensure they're compatible across
all implementations of Jinja:

- Replace Python methods with Jinja filters. These usually have the same name, for example `string.lower()` becomes
  `string|lower`, and `dict.items()` becomes `dict|items`. One notable change is that `string.strip()` becomes `string|trim`.
  See the [list of built-in filters](https://jinja.palletsprojects.com/en/3.1.x/templates/#builtin-filters)
  in the Jinja documentation for more.
- Replace `True`, `False` and `None`, which are Python-specific, with `true`, `false` and `none`.
- Directly rendering a dict or list may give different results in other implementations (for example, string entries
  might change from single-quoted to double-quoted). Adding the `tojson` filter can help to ensure consistency here.


key: _node_content
string_value:
{"id_": "e09ed79f-1c91-4cac-80f8-30248477b485", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d58a77e8-5ea6-4ec1-988a-8b8566c60a0b", "node_type": "1", "metadata": {}, "hash": "b226a18b18a0ff2715c0422af7dd5128eb99b351bdf1dbb5c8c2fce313f98236", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3234, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to add a model to 🤗 Transformers?

The 🤗 Transformers library is often able to offer new models thanks to community contributors. But this can be a challenging project and requires an in-depth knowledge of the 🤗 Transformers library and the model to implement. At Hugging Face, we're trying to empower more of the community to actively add models and we've put together this guide to walk you through the process of adding a PyTorch model (make sure you have [PyTorch installed](https://pytorch.org/get-started/locally/)).

Along the way, you'll:

- get insights into open-source best practices
- understand the design principles behind one of the most popular deep learning libraries
- learn how to efficiently test large models
- learn how to integrate Python utilities like `black`, `ruff`, and `make fix-copies` to ensure clean and readable code

A Hugging Face team member will be available to help you along the way so you'll never be alone. 🤗 ❤️

To get started, open a [New model addition](https://github.com/huggingface/transformers/issues/new?assignees=&labels=New+model&template=new-model-addition.yml) issue for the model you want to see in 🤗 Transformers. If you're not especially picky about contributing a specific model, you can filter by the [New model label](https://github.com/huggingface/transformers/labels/New%20model) to see if there are any unclaimed model requests and work on it.

Once you've opened a new model request, the first step is to get familiar with 🤗 Transformers if you aren't already!

## General overview of 🤗 Transformers

First, you should get a general overview of 🤗 Transformers. 🤗 Transformers is a very opinionated library, so there is a
chance that you don't agree with some of the library's philosophies or design choices. From our experience, however, we
found that the fundamental design choices and philosophies of the library are crucial to efficiently scale 🤗
Transformers while keeping maintenance costs at a reasonable level.

A good first starting point to better understand the library is to read the [documentation of our philosophy](philosophy). As a result of our way of working, there are some choices that we try to apply to all models:

- Composition is generally favored over-abstraction
- Duplicating code is not always bad if it strongly improves the readability or accessibility of a model
- Model files are as self-contained as possible so that when you read the code of a specific model, you ideally only
  have to look into the respective `modeling_....py` file.

In our opinion, the library's code is not just a means to provide a product, *e.g.* the ability to use BERT for
inference, but also as the very product that we want to improve. Hence, when adding a model, the user is not only the
person who will use your model, but also everybody who will read, try to understand, and possibly tweak your code.

With this in mind, let's go a bit deeper into the general library design.

### Overview of models

To successfully add a model, it is important to understand the interaction between your model and its config,
[`PreTrainedModel`], and [`PretrainedConfig`]. For exemplary purposes, we will
call the model to be added to 🤗 Transformers `BrandNewBert`.


key: _node_content
string_value:
{"id_": "d58a77e8-5ea6-4ec1-988a-8b8566c60a0b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e09ed79f-1c91-4cac-80f8-30248477b485", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "9fb421124cef8f0c531b3296d23d73749bc33d19139382c97cde2be697555702", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "40f86fda-ee6f-4227-b96f-03abb0727b0d", "node_type": "1", "metadata": {}, "hash": "55e340fc7b813885869bdc0a62d5f54ce62ce1b90e1ffc6c2fece981c6c0e784", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1531, "end_char_idx": 4527, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## General overview of 🤗 Transformers

First, you should get a general overview of 🤗 Transformers. 🤗 Transformers is a very opinionated library, so there is a
chance that you don't agree with some of the library's philosophies or design choices. From our experience, however, we
found that the fundamental design choices and philosophies of the library are crucial to efficiently scale 🤗
Transformers while keeping maintenance costs at a reasonable level.

A good first starting point to better understand the library is to read the [documentation of our philosophy](philosophy). As a result of our way of working, there are some choices that we try to apply to all models:

- Composition is generally favored over-abstraction
- Duplicating code is not always bad if it strongly improves the readability or accessibility of a model
- Model files are as self-contained as possible so that when you read the code of a specific model, you ideally only
  have to look into the respective `modeling_....py` file.

In our opinion, the library's code is not just a means to provide a product, *e.g.* the ability to use BERT for
inference, but also as the very product that we want to improve. Hence, when adding a model, the user is not only the
person who will use your model, but also everybody who will read, try to understand, and possibly tweak your code.

With this in mind, let's go a bit deeper into the general library design.

### Overview of models

To successfully add a model, it is important to understand the interaction between your model and its config,
[`PreTrainedModel`], and [`PretrainedConfig`]. For exemplary purposes, we will
call the model to be added to 🤗 Transformers `BrandNewBert`.

Let's take a look:

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers_overview.png"/>

As you can see, we do make use of inheritance in 🤗 Transformers, but we keep the level of abstraction to an absolute
minimum. There are never more than two levels of abstraction for any model in the library. `BrandNewBertModel`
inherits from `BrandNewBertPreTrainedModel` which in turn inherits from [`PreTrainedModel`] and
that's it. As a general rule, we want to make sure that a new model only depends on
[`PreTrainedModel`]. The important functionalities that are automatically provided to every new
model are [`~PreTrainedModel.from_pretrained`] and
[`~PreTrainedModel.save_pretrained`], which are used for serialization and deserialization. All of the
other important functionalities, such as `BrandNewBertModel.forward` should be completely defined in the new
`modeling_brand_new_bert.py` script. Next, we want to make sure that a model with a specific head layer, such as
`BrandNewBertForMaskedLM` does not inherit from `BrandNewBertModel`, but rather uses `BrandNewBertModel`
as a component that can be called in its forward pass to keep the level of abstraction low. Every new model requires a
configuration class, called `BrandNewBertConfig`.


key: _node_content
string_value:
{"id_": "40f86fda-ee6f-4227-b96f-03abb0727b0d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d58a77e8-5ea6-4ec1-988a-8b8566c60a0b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "599d4cb4920a032486754f52278809d177cab7546e7ccd0005e7afb96865be20", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d5cd700-5f79-4a87-b1e9-1205338c1439", "node_type": "1", "metadata": {}, "hash": "db2e62acb5a2af71f77dce749b331e11674c5a3f6f81328e01718c3a19140510", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2886, "end_char_idx": 6173, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With this in mind, let's go a bit deeper into the general library design.

### Overview of models

To successfully add a model, it is important to understand the interaction between your model and its config,
[`PreTrainedModel`], and [`PretrainedConfig`]. For exemplary purposes, we will
call the model to be added to 🤗 Transformers `BrandNewBert`.

Let's take a look:

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers_overview.png"/>

As you can see, we do make use of inheritance in 🤗 Transformers, but we keep the level of abstraction to an absolute
minimum. There are never more than two levels of abstraction for any model in the library. `BrandNewBertModel`
inherits from `BrandNewBertPreTrainedModel` which in turn inherits from [`PreTrainedModel`] and
that's it. As a general rule, we want to make sure that a new model only depends on
[`PreTrainedModel`]. The important functionalities that are automatically provided to every new
model are [`~PreTrainedModel.from_pretrained`] and
[`~PreTrainedModel.save_pretrained`], which are used for serialization and deserialization. All of the
other important functionalities, such as `BrandNewBertModel.forward` should be completely defined in the new
`modeling_brand_new_bert.py` script. Next, we want to make sure that a model with a specific head layer, such as
`BrandNewBertForMaskedLM` does not inherit from `BrandNewBertModel`, but rather uses `BrandNewBertModel`
as a component that can be called in its forward pass to keep the level of abstraction low. Every new model requires a
configuration class, called `BrandNewBertConfig`. This configuration is always stored as an attribute in
[`PreTrainedModel`], and thus can be accessed via the `config` attribute for all classes
inheriting from `BrandNewBertPreTrainedModel`:

```python
model = BrandNewBertModel.from_pretrained("brandy/brand_new_bert")
model.config  # model has access to its config
```

Similar to the model, the configuration inherits basic serialization and deserialization functionalities from
[`PretrainedConfig`]. Note that the configuration and the model are always serialized into two
different formats - the model to a *pytorch_model.bin* file and the configuration to a *config.json* file. Calling
the model's [`~PreTrainedModel.save_pretrained`] will automatically call
the config's [`~PretrainedConfig.save_pretrained`], so that both model and configuration are saved.


### Code style

When coding your new model, keep in mind that Transformers is an opinionated library and we have a few quirks of our
own regarding how code should be written :-)

1. The forward pass of your model should be fully written in the modeling file while being fully independent of other
   models in the library. If you want to reuse a block from another model, copy the code and paste it with a
   `# Copied from` comment on top (see [here](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/models/roberta/modeling_roberta.py#L160)
   for a good example and [there](pr_checks#check-copies) for more documentation on Copied from). 
2. The code should be fully understandable, even by a non-native English speaker. This means you should pick
   descriptive variable names and avoid abbreviations.


key: _node_content
string_value:
{"id_": "4d5cd700-5f79-4a87-b1e9-1205338c1439", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "40f86fda-ee6f-4227-b96f-03abb0727b0d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "8dffee4b0209628a0344fbbe9c6e72d3c0e2fefa62190277169a323e00892b15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3f14874a-6d9b-4660-b2aa-b42ca6fd4cc6", "node_type": "1", "metadata": {}, "hash": "aebd6107d9b984589933cf8bb5c321d05161a68552ef15f02410b8268f3e1539", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4528, "end_char_idx": 7719, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This configuration is always stored as an attribute in
[`PreTrainedModel`], and thus can be accessed via the `config` attribute for all classes
inheriting from `BrandNewBertPreTrainedModel`:

```python
model = BrandNewBertModel.from_pretrained("brandy/brand_new_bert")
model.config  # model has access to its config
```

Similar to the model, the configuration inherits basic serialization and deserialization functionalities from
[`PretrainedConfig`]. Note that the configuration and the model are always serialized into two
different formats - the model to a *pytorch_model.bin* file and the configuration to a *config.json* file. Calling
the model's [`~PreTrainedModel.save_pretrained`] will automatically call
the config's [`~PretrainedConfig.save_pretrained`], so that both model and configuration are saved.


### Code style

When coding your new model, keep in mind that Transformers is an opinionated library and we have a few quirks of our
own regarding how code should be written :-)

1. The forward pass of your model should be fully written in the modeling file while being fully independent of other
   models in the library. If you want to reuse a block from another model, copy the code and paste it with a
   `# Copied from` comment on top (see [here](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/models/roberta/modeling_roberta.py#L160)
   for a good example and [there](pr_checks#check-copies) for more documentation on Copied from). 
2. The code should be fully understandable, even by a non-native English speaker. This means you should pick
   descriptive variable names and avoid abbreviations. As an example, `activation` is preferred to `act`.
   One-letter variable names are strongly discouraged unless it's an index in a for loop.
3. More generally we prefer longer explicit code to short magical one.
4. Avoid subclassing `nn.Sequential` in PyTorch but subclass `nn.Module` and write the forward pass, so that anyone
   using your code can quickly debug it by adding print statements or breaking points.
5. Your function signature should be type-annotated. For the rest, good variable names are way more readable and
   understandable than type annotations.

### Overview of tokenizers

Not quite ready yet :-( This section will be added soon!

## Step-by-step recipe to add a model to 🤗 Transformers

Everyone has different preferences of how to port a model so it can be very helpful for you to take a look at summaries
of how other contributors ported models to Hugging Face. Here is a list of community blog posts on how to port a model:

1. [Porting GPT2 Model](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28) by [Thomas](https://huggingface.co/thomwolf)
2. [Porting WMT19 MT Model](https://huggingface.co/blog/porting-fsmt) by [Stas](https://huggingface.co/stas)

From experience, we can tell you that the most important things to keep in mind when adding a model are:

-  Don't reinvent the wheel! Most parts of the code you will add for the new 🤗 Transformers model already exist
  somewhere in 🤗 Transformers. Take some time to find similar, already existing models and tokenizers you can copy
  from.


key: _node_content
string_value:
{"id_": "3f14874a-6d9b-4660-b2aa-b42ca6fd4cc6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4d5cd700-5f79-4a87-b1e9-1205338c1439", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "efcadc0aba4f766d6b9d518f95092c142e69d9a44ef09a72cea5ef5c91eb86cb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "685e6d2b-53b9-4677-b9b3-914889880efd", "node_type": "1", "metadata": {}, "hash": "c236486c6ce9da7fa02c53e36b0c14b99062c69816a971f539be47e6c58dc569", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6092, "end_char_idx": 8662, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means you should pick
   descriptive variable names and avoid abbreviations. As an example, `activation` is preferred to `act`.
   One-letter variable names are strongly discouraged unless it's an index in a for loop.
3. More generally we prefer longer explicit code to short magical one.
4. Avoid subclassing `nn.Sequential` in PyTorch but subclass `nn.Module` and write the forward pass, so that anyone
   using your code can quickly debug it by adding print statements or breaking points.
5. Your function signature should be type-annotated. For the rest, good variable names are way more readable and
   understandable than type annotations.

### Overview of tokenizers

Not quite ready yet :-( This section will be added soon!

## Step-by-step recipe to add a model to 🤗 Transformers

Everyone has different preferences of how to port a model so it can be very helpful for you to take a look at summaries
of how other contributors ported models to Hugging Face. Here is a list of community blog posts on how to port a model:

1. [Porting GPT2 Model](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28) by [Thomas](https://huggingface.co/thomwolf)
2. [Porting WMT19 MT Model](https://huggingface.co/blog/porting-fsmt) by [Stas](https://huggingface.co/stas)

From experience, we can tell you that the most important things to keep in mind when adding a model are:

-  Don't reinvent the wheel! Most parts of the code you will add for the new 🤗 Transformers model already exist
  somewhere in 🤗 Transformers. Take some time to find similar, already existing models and tokenizers you can copy
  from. [grep](https://www.gnu.org/software/grep/) and [rg](https://github.com/BurntSushi/ripgrep) are your
  friends. Note that it might very well happen that your model's tokenizer is based on one model implementation, and
  your model's modeling code on another one. *E.g.* FSMT's modeling code is based on BART, while FSMT's tokenizer code
  is based on XLM.
-  It's more of an engineering challenge than a scientific challenge. You should spend more time creating an
  efficient debugging environment rather than trying to understand all theoretical aspects of the model in the paper.
-  Ask for help, when you're stuck! Models are the core component of 🤗 Transformers so we at Hugging Face are more
  than happy to help you at every step to add your model. Don't hesitate to ask if you notice you are not making
  progress.

In the following, we try to give you a general recipe that we found most useful when porting a model to 🤗 Transformers.


key: _node_content
string_value:
{"id_": "685e6d2b-53b9-4677-b9b3-914889880efd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3f14874a-6d9b-4660-b2aa-b42ca6fd4cc6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "85ee1b28729c1ab320a1c6423134bb1cad525cd2b58c01dbb959dcc684614600", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aae5bb19-b3d6-40c1-b4d8-7701a6c5ed5f", "node_type": "1", "metadata": {}, "hash": "af9de8a25317467879b8e1cc32a798ce305c8c10626afd614c2ab9f35bb60a8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7131, "end_char_idx": 10359, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Porting GPT2 Model](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28) by [Thomas](https://huggingface.co/thomwolf)
2. [Porting WMT19 MT Model](https://huggingface.co/blog/porting-fsmt) by [Stas](https://huggingface.co/stas)

From experience, we can tell you that the most important things to keep in mind when adding a model are:

-  Don't reinvent the wheel! Most parts of the code you will add for the new 🤗 Transformers model already exist
  somewhere in 🤗 Transformers. Take some time to find similar, already existing models and tokenizers you can copy
  from. [grep](https://www.gnu.org/software/grep/) and [rg](https://github.com/BurntSushi/ripgrep) are your
  friends. Note that it might very well happen that your model's tokenizer is based on one model implementation, and
  your model's modeling code on another one. *E.g.* FSMT's modeling code is based on BART, while FSMT's tokenizer code
  is based on XLM.
-  It's more of an engineering challenge than a scientific challenge. You should spend more time creating an
  efficient debugging environment rather than trying to understand all theoretical aspects of the model in the paper.
-  Ask for help, when you're stuck! Models are the core component of 🤗 Transformers so we at Hugging Face are more
  than happy to help you at every step to add your model. Don't hesitate to ask if you notice you are not making
  progress.

In the following, we try to give you a general recipe that we found most useful when porting a model to 🤗 Transformers.

The following list is a summary of everything that has to be done to add a model and can be used by you as a To-Do
List:

☐ (Optional) Understood the model's theoretical aspects<br>
☐ Prepared 🤗 Transformers dev environment<br>
☐ Set up debugging environment of the original repository<br>
☐ Created script that successfully runs the `forward()` pass using the original repository and checkpoint<br>
☐ Successfully added the model skeleton to 🤗 Transformers<br>
☐ Successfully converted original checkpoint to 🤗 Transformers checkpoint<br>
☐ Successfully ran `forward()` pass in 🤗 Transformers that gives identical output to original checkpoint<br>
☐ Finished model tests in 🤗 Transformers<br>
☐ Successfully added tokenizer in 🤗 Transformers<br>
☐ Run end-to-end integration tests<br>
☐ Finished docs<br>
☐ Uploaded model weights to the Hub<br>
☐ Submitted the pull request<br>
☐ (Optional) Added a demo notebook

To begin with, we usually recommend starting by getting a good theoretical understanding of `BrandNewBert`. However,
if you prefer to understand the theoretical aspects of the model *on-the-job*, then it is totally fine to directly dive
into the `BrandNewBert`'s code-base. This option might suit you better if your engineering skills are better than
your theoretical skill, if you have trouble understanding `BrandNewBert`'s paper, or if you just enjoy programming
much more than reading scientific papers.

### 1. (Optional) Theoretical aspects of BrandNewBert

You should take some time to read *BrandNewBert's* paper, if such descriptive work exists. There might be large
sections of the paper that are difficult to understand. If this is the case, this is fine - don't worry!


key: _node_content
string_value:
{"id_": "aae5bb19-b3d6-40c1-b4d8-7701a6c5ed5f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "685e6d2b-53b9-4677-b9b3-914889880efd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "6353f44da589d0c2fbfcb84e7c2ee7a2c78ac1d05b2759e14e683f7178d08244", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "80885097-dab2-4eb6-982d-2b0cefd29b9c", "node_type": "1", "metadata": {}, "hash": "948af9405745c9f24d617e415809f2cfbd55ec33e9c9e52eac5b8fb3add1476d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8664, "end_char_idx": 11945, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The following list is a summary of everything that has to be done to add a model and can be used by you as a To-Do
List:

☐ (Optional) Understood the model's theoretical aspects<br>
☐ Prepared 🤗 Transformers dev environment<br>
☐ Set up debugging environment of the original repository<br>
☐ Created script that successfully runs the `forward()` pass using the original repository and checkpoint<br>
☐ Successfully added the model skeleton to 🤗 Transformers<br>
☐ Successfully converted original checkpoint to 🤗 Transformers checkpoint<br>
☐ Successfully ran `forward()` pass in 🤗 Transformers that gives identical output to original checkpoint<br>
☐ Finished model tests in 🤗 Transformers<br>
☐ Successfully added tokenizer in 🤗 Transformers<br>
☐ Run end-to-end integration tests<br>
☐ Finished docs<br>
☐ Uploaded model weights to the Hub<br>
☐ Submitted the pull request<br>
☐ (Optional) Added a demo notebook

To begin with, we usually recommend starting by getting a good theoretical understanding of `BrandNewBert`. However,
if you prefer to understand the theoretical aspects of the model *on-the-job*, then it is totally fine to directly dive
into the `BrandNewBert`'s code-base. This option might suit you better if your engineering skills are better than
your theoretical skill, if you have trouble understanding `BrandNewBert`'s paper, or if you just enjoy programming
much more than reading scientific papers.

### 1. (Optional) Theoretical aspects of BrandNewBert

You should take some time to read *BrandNewBert's* paper, if such descriptive work exists. There might be large
sections of the paper that are difficult to understand. If this is the case, this is fine - don't worry! The goal is
not to get a deep theoretical understanding of the paper, but to extract the necessary information required to
effectively re-implement the model in 🤗 Transformers. That being said, you don't have to spend too much time on the
theoretical aspects, but rather focus on the practical ones, namely:

-  What type of model is *brand_new_bert*? BERT-like encoder-only model? GPT2-like decoder-only model? BART-like
  encoder-decoder model? Look at the [model_summary](model_summary) if you're not familiar with the differences between those.
-  What are the applications of *brand_new_bert*? Text classification? Text generation? Seq2Seq tasks, *e.g.,*
  summarization?
-  What is the novel feature of the model that makes it different from BERT/GPT-2/BART?
-  Which of the already existing [🤗 Transformers models](https://huggingface.co/transformers/#contents) is most
  similar to *brand_new_bert*?
-  What type of tokenizer is used? A sentencepiece tokenizer? Word piece tokenizer? Is it the same tokenizer as used
  for BERT or BART?

After you feel like you have gotten a good overview of the architecture of the model, you might want to write to the
Hugging Face team with any questions you might have. This might include questions regarding the model's architecture,
its attention layer, etc. We will be more than happy to help you.

### 2. Next prepare your environment

1. Fork the [repository](https://github.com/huggingface/transformers) by clicking on the ‘Fork' button on the
   repository's page. This creates a copy of the code under your GitHub user account.

2.


key: _node_content
string_value:
{"id_": "80885097-dab2-4eb6-982d-2b0cefd29b9c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aae5bb19-b3d6-40c1-b4d8-7701a6c5ed5f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "2fac60ef1bb3fbc91fe99cde393329444534c62e92f21744733b4bdb802a637d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0d44b683-5ef8-4991-86df-3dcc8d57601e", "node_type": "1", "metadata": {}, "hash": "9aaf36e639c7816d40b8eef8fdb0d0c60da77962cd551f7e49fd55fde9a094c1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10360, "end_char_idx": 13466, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The goal is
not to get a deep theoretical understanding of the paper, but to extract the necessary information required to
effectively re-implement the model in 🤗 Transformers. That being said, you don't have to spend too much time on the
theoretical aspects, but rather focus on the practical ones, namely:

-  What type of model is *brand_new_bert*? BERT-like encoder-only model? GPT2-like decoder-only model? BART-like
  encoder-decoder model? Look at the [model_summary](model_summary) if you're not familiar with the differences between those.
-  What are the applications of *brand_new_bert*? Text classification? Text generation? Seq2Seq tasks, *e.g.,*
  summarization?
-  What is the novel feature of the model that makes it different from BERT/GPT-2/BART?
-  Which of the already existing [🤗 Transformers models](https://huggingface.co/transformers/#contents) is most
  similar to *brand_new_bert*?
-  What type of tokenizer is used? A sentencepiece tokenizer? Word piece tokenizer? Is it the same tokenizer as used
  for BERT or BART?

After you feel like you have gotten a good overview of the architecture of the model, you might want to write to the
Hugging Face team with any questions you might have. This might include questions regarding the model's architecture,
its attention layer, etc. We will be more than happy to help you.

### 2. Next prepare your environment

1. Fork the [repository](https://github.com/huggingface/transformers) by clicking on the ‘Fork' button on the
   repository's page. This creates a copy of the code under your GitHub user account.

2. Clone your `transformers` fork to your local disk, and add the base repository as a remote:

   ```bash
   git clone https://github.com/[your Github handle]/transformers.git
   cd transformers
   git remote add upstream https://github.com/huggingface/transformers.git
   ```

3. Set up a development environment, for instance by running the following command:

   ```bash
   python -m venv .env
   source .env/bin/activate
   pip install -e ".[dev]"
   ```

   Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a
   failure with this command. If that's the case make sure to install the Deep Learning framework you are working with
   (PyTorch, TensorFlow and/or Flax) then do:

   ```bash
   pip install -e ".[quality]"
   ```

   which should be enough for most use cases. You can then return to the parent directory

   ```bash
   cd ..
   ```

4. We recommend adding the PyTorch version of *brand_new_bert* to Transformers. To install PyTorch, please follow the
   instructions on https://pytorch.org/get-started/locally/.

   **Note:** You don't need to have CUDA installed. Making the new model work on CPU is sufficient.

5. To port *brand_new_bert*, you will also need access to its original repository:

   ```bash
   git clone https://github.com/org_that_created_brand_new_bert_org/brand_new_bert.git
   cd brand_new_bert
   pip install -e .
   ```

Now you have set up a development environment to port *brand_new_bert* to 🤗 Transformers.

### 3.-4.


key: _node_content
string_value:
{"id_": "0d44b683-5ef8-4991-86df-3dcc8d57601e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "80885097-dab2-4eb6-982d-2b0cefd29b9c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "5a8a41d87fc34e6c79f100d3bf25cb9aacd7079f54e017ab85f663bb37975914", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "88ed9cb4-6412-4b2a-88d2-3a281e12f349", "node_type": "1", "metadata": {}, "hash": "0349d8d3844d8364eac2e77a6c75e52a4af6bbcbb068e8172aa1b0e6fc1b9d54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11943, "end_char_idx": 15221, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. Clone your `transformers` fork to your local disk, and add the base repository as a remote:

   ```bash
   git clone https://github.com/[your Github handle]/transformers.git
   cd transformers
   git remote add upstream https://github.com/huggingface/transformers.git
   ```

3. Set up a development environment, for instance by running the following command:

   ```bash
   python -m venv .env
   source .env/bin/activate
   pip install -e ".[dev]"
   ```

   Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a
   failure with this command. If that's the case make sure to install the Deep Learning framework you are working with
   (PyTorch, TensorFlow and/or Flax) then do:

   ```bash
   pip install -e ".[quality]"
   ```

   which should be enough for most use cases. You can then return to the parent directory

   ```bash
   cd ..
   ```

4. We recommend adding the PyTorch version of *brand_new_bert* to Transformers. To install PyTorch, please follow the
   instructions on https://pytorch.org/get-started/locally/.

   **Note:** You don't need to have CUDA installed. Making the new model work on CPU is sufficient.

5. To port *brand_new_bert*, you will also need access to its original repository:

   ```bash
   git clone https://github.com/org_that_created_brand_new_bert_org/brand_new_bert.git
   cd brand_new_bert
   pip install -e .
   ```

Now you have set up a development environment to port *brand_new_bert* to 🤗 Transformers.

### 3.-4. Run a pretrained checkpoint using the original repository

At first, you will work on the original *brand_new_bert* repository. Often, the original implementation is very
“researchy”. Meaning that documentation might be lacking and the code can be difficult to understand. But this should
be exactly your motivation to reimplement *brand_new_bert*. At Hugging Face, one of our main goals is to *make people
stand on the shoulders of giants* which translates here very well into taking a working model and rewriting it to make
it as **accessible, user-friendly, and beautiful** as possible. This is the number-one motivation to re-implement
models into 🤗 Transformers - trying to make complex new NLP technology accessible to **everybody**.

You should start thereby by diving into the original repository.

Successfully running the official pretrained model in the original repository is often **the most difficult** step.
From our experience, it is very important to spend some time getting familiar with the original code-base. You need to
figure out the following:

- Where to find the pretrained weights?
- How to load the pretrained weights into the corresponding model?
- How to run the tokenizer independently from the model?
- Trace one forward pass so that you know which classes and functions are required for a simple forward pass. Usually,
  you only have to reimplement those functions.
- Be able to locate the important components of the model: Where is the model's class? Are there model sub-classes,
  *e.g.* EncoderModel, DecoderModel? Where is the self-attention layer? Are there multiple different attention layers,
  *e.g.* *self-attention*, *cross-attention*...?
- How can you debug the model in the original environment of the repo?


key: _node_content
string_value:
{"id_": "88ed9cb4-6412-4b2a-88d2-3a281e12f349", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0d44b683-5ef8-4991-86df-3dcc8d57601e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "6780076fc18998b851cc70a5773957296d8892ca2057df54effde0563728ab6a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ccbbb71-0cdf-4ca2-a53b-3acadfbfcf18", "node_type": "1", "metadata": {}, "hash": "8b70049ff604f8f5d1e707e669783f73920110b664ecfbd0eb610b5e53a2b950", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13457, "end_char_idx": 16973, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### 3.-4. Run a pretrained checkpoint using the original repository

At first, you will work on the original *brand_new_bert* repository. Often, the original implementation is very
“researchy”. Meaning that documentation might be lacking and the code can be difficult to understand. But this should
be exactly your motivation to reimplement *brand_new_bert*. At Hugging Face, one of our main goals is to *make people
stand on the shoulders of giants* which translates here very well into taking a working model and rewriting it to make
it as **accessible, user-friendly, and beautiful** as possible. This is the number-one motivation to re-implement
models into 🤗 Transformers - trying to make complex new NLP technology accessible to **everybody**.

You should start thereby by diving into the original repository.

Successfully running the official pretrained model in the original repository is often **the most difficult** step.
From our experience, it is very important to spend some time getting familiar with the original code-base. You need to
figure out the following:

- Where to find the pretrained weights?
- How to load the pretrained weights into the corresponding model?
- How to run the tokenizer independently from the model?
- Trace one forward pass so that you know which classes and functions are required for a simple forward pass. Usually,
  you only have to reimplement those functions.
- Be able to locate the important components of the model: Where is the model's class? Are there model sub-classes,
  *e.g.* EncoderModel, DecoderModel? Where is the self-attention layer? Are there multiple different attention layers,
  *e.g.* *self-attention*, *cross-attention*...?
- How can you debug the model in the original environment of the repo? Do you have to add *print* statements, can you
  work with an interactive debugger like *ipdb*, or should you use an efficient IDE to debug the model, like PyCharm?

It is very important that before you start the porting process, you can **efficiently** debug code in the original
repository! Also, remember that you are working with an open-source library, so do not hesitate to open an issue, or
even a pull request in the original repository. The maintainers of this repository are most likely very happy about
someone looking into their code!

At this point, it is really up to you which debugging environment and strategy you prefer to use to debug the original
model. We strongly advise against setting up a costly GPU environment, but simply work on a CPU both when starting to
dive into the original repository and also when starting to write the 🤗 Transformers implementation of the model. Only
at the very end, when the model has already been successfully ported to 🤗 Transformers, one should verify that the
model also works as expected on GPU.

In general, there are two possible debugging environments for running the original model

-  [Jupyter notebooks](https://jupyter.org/) / [google colab](https://colab.research.google.com/notebooks/intro.ipynb)
-  Local python scripts.

Jupyter notebooks have the advantage that they allow for cell-by-cell execution which can be helpful to better split
logical components from one another and to have faster debugging cycles as intermediate results can be stored. Also,
notebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging
Face team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them.


key: _node_content
string_value:
{"id_": "2ccbbb71-0cdf-4ca2-a53b-3acadfbfcf18", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "88ed9cb4-6412-4b2a-88d2-3a281e12f349", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "dd05effb5d9216329253572d6d1cffec7b63f0f9f895b582d6503f0b47531030", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e70cdbc-55c0-4249-a9fe-c607453b2c7c", "node_type": "1", "metadata": {}, "hash": "ae1f10527656926adfd0bd2fab62ba4aa1cd3f58df2f7edfea75b640d7b87db6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15151, "end_char_idx": 18407, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- How can you debug the model in the original environment of the repo? Do you have to add *print* statements, can you
  work with an interactive debugger like *ipdb*, or should you use an efficient IDE to debug the model, like PyCharm?

It is very important that before you start the porting process, you can **efficiently** debug code in the original
repository! Also, remember that you are working with an open-source library, so do not hesitate to open an issue, or
even a pull request in the original repository. The maintainers of this repository are most likely very happy about
someone looking into their code!

At this point, it is really up to you which debugging environment and strategy you prefer to use to debug the original
model. We strongly advise against setting up a costly GPU environment, but simply work on a CPU both when starting to
dive into the original repository and also when starting to write the 🤗 Transformers implementation of the model. Only
at the very end, when the model has already been successfully ported to 🤗 Transformers, one should verify that the
model also works as expected on GPU.

In general, there are two possible debugging environments for running the original model

-  [Jupyter notebooks](https://jupyter.org/) / [google colab](https://colab.research.google.com/notebooks/intro.ipynb)
-  Local python scripts.

Jupyter notebooks have the advantage that they allow for cell-by-cell execution which can be helpful to better split
logical components from one another and to have faster debugging cycles as intermediate results can be stored. Also,
notebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging
Face team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them.

The obvious disadvantage of Jupyter notebooks is that if you are not used to working with them you will have to spend
some time adjusting to the new programming environment and you might not be able to use your known debugging tools
anymore, like `ipdb`.

For each code-base, a good first step is always to load a **small** pretrained checkpoint and to be able to reproduce a
single forward pass using a dummy integer vector of input IDs as an input. Such a script could look like this (in
pseudocode):

```python
model = BrandNewBertModel.load_pretrained_checkpoint("/path/to/checkpoint/")
input_ids = [0, 4, 5, 2, 3, 7, 9]  # vector of input ids
original_output = model.predict(input_ids)
```

Next, regarding the debugging strategy, there are generally a few from which to choose from:

- Decompose the original model into many small testable components and run a forward pass on each of those for
  verification
- Decompose the original model only into the original *tokenizer* and the original *model*, run a forward pass on
  those, and use intermediate print statements or breakpoints for verification

Again, it is up to you which strategy to choose. Often, one or the other is advantageous depending on the original code
base.

If the original code-base allows you to decompose the model into smaller sub-components, *e.g.* if the original
code-base can easily be run in eager mode, it is usually worth the effort to do so.


key: _node_content
string_value:
{"id_": "2e70cdbc-55c0-4249-a9fe-c607453b2c7c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ccbbb71-0cdf-4ca2-a53b-3acadfbfcf18", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "321653e0f8b1395e19610df2d871d32b4a7d5c6a4488c856923333ceffd663b2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f5b4c0b5-98f1-4482-86a7-c6e152a48b07", "node_type": "1", "metadata": {}, "hash": "a736c2ca4d0fdfeec83a5fc5a01243831daeabceef8f2c8e58b908fc4bdce857", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16742, "end_char_idx": 20335, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Also,
notebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging
Face team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them.

The obvious disadvantage of Jupyter notebooks is that if you are not used to working with them you will have to spend
some time adjusting to the new programming environment and you might not be able to use your known debugging tools
anymore, like `ipdb`.

For each code-base, a good first step is always to load a **small** pretrained checkpoint and to be able to reproduce a
single forward pass using a dummy integer vector of input IDs as an input. Such a script could look like this (in
pseudocode):

```python
model = BrandNewBertModel.load_pretrained_checkpoint("/path/to/checkpoint/")
input_ids = [0, 4, 5, 2, 3, 7, 9]  # vector of input ids
original_output = model.predict(input_ids)
```

Next, regarding the debugging strategy, there are generally a few from which to choose from:

- Decompose the original model into many small testable components and run a forward pass on each of those for
  verification
- Decompose the original model only into the original *tokenizer* and the original *model*, run a forward pass on
  those, and use intermediate print statements or breakpoints for verification

Again, it is up to you which strategy to choose. Often, one or the other is advantageous depending on the original code
base.

If the original code-base allows you to decompose the model into smaller sub-components, *e.g.* if the original
code-base can easily be run in eager mode, it is usually worth the effort to do so. There are some important advantages
to taking the more difficult road in the beginning:

- at a later stage when comparing the original model to the Hugging Face implementation, you can verify automatically
  for each component individually that the corresponding component of the 🤗 Transformers implementation matches instead
  of relying on visual comparison via print statements
- it can give you some rope to decompose the big problem of porting a model into smaller problems of just porting
  individual components and thus structure your work better
- separating the model into logical meaningful components will help you to get a better overview of the model's design
  and thus to better understand the model
- at a later stage those component-by-component tests help you to ensure that no regression occurs as you continue
  changing your code

[Lysandre's](https://gist.github.com/LysandreJik/db4c948f6b4483960de5cbac598ad4ed) integration checks for ELECTRA
gives a nice example of how this can be done.

However, if the original code-base is very complex or only allows intermediate components to be run in a compiled mode,
it might be too time-consuming or even impossible to separate the model into smaller testable sub-components. A good
example is [T5's MeshTensorFlow](https://github.com/tensorflow/mesh/tree/master/mesh_tensorflow) library which is
very complex and does not offer a simple way to decompose the model into its sub-components. For such libraries, one
often relies on verifying print statements.

No matter which strategy you choose, the recommended procedure is often the same that you should start to debug the
starting layers first and the ending layers last.

It is recommended that you retrieve the output, either by print statements or sub-component functions, of the following
layers in the following order:

1. Retrieve the input IDs passed to the model
2. Retrieve the word embeddings
3.


key: _node_content
string_value:
{"id_": "f5b4c0b5-98f1-4482-86a7-c6e152a48b07", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e70cdbc-55c0-4249-a9fe-c607453b2c7c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "cde417571e1b6facd61edf08699322359edbe83ab4a20ee473e59733c8733024", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "594bc362-1b95-4fb8-b421-79738ccfe2a7", "node_type": "1", "metadata": {}, "hash": "fb0c9573da64959e8a525f2ee61e1e6326cf6c6c04dc8cc78370563f589f92ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19423, "end_char_idx": 22179, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, if the original code-base is very complex or only allows intermediate components to be run in a compiled mode,
it might be too time-consuming or even impossible to separate the model into smaller testable sub-components. A good
example is [T5's MeshTensorFlow](https://github.com/tensorflow/mesh/tree/master/mesh_tensorflow) library which is
very complex and does not offer a simple way to decompose the model into its sub-components. For such libraries, one
often relies on verifying print statements.

No matter which strategy you choose, the recommended procedure is often the same that you should start to debug the
starting layers first and the ending layers last.

It is recommended that you retrieve the output, either by print statements or sub-component functions, of the following
layers in the following order:

1. Retrieve the input IDs passed to the model
2. Retrieve the word embeddings
3. Retrieve the input of the first Transformer layer
4. Retrieve the output of the first Transformer layer
5. Retrieve the output of the following n - 1 Transformer layers
6. Retrieve the output of the whole BrandNewBert Model

Input IDs should thereby consists of an array of integers, *e.g.* `input_ids = [0, 4, 4, 3, 2, 4, 1, 7, 19]`

The outputs of the following layers often consist of multi-dimensional float arrays and can look like this:

```
[[
 [-0.1465, -0.6501,  0.1993,  ...,  0.1451,  0.3430,  0.6024],
 [-0.4417, -0.5920,  0.3450,  ..., -0.3062,  0.6182,  0.7132],
 [-0.5009, -0.7122,  0.4548,  ..., -0.3662,  0.6091,  0.7648],
 ...,
 [-0.5613, -0.6332,  0.4324,  ..., -0.3792,  0.7372,  0.9288],
 [-0.5416, -0.6345,  0.4180,  ..., -0.3564,  0.6992,  0.9191],
 [-0.5334, -0.6403,  0.4271,  ..., -0.3339,  0.6533,  0.8694]]],
```

We expect that every model added to 🤗 Transformers passes a couple of integration tests, meaning that the original
model and the reimplemented version in 🤗 Transformers have to give the exact same output up to a precision of 0.001!
Since it is normal that the exact same model written in different libraries can give a slightly different output
depending on the library framework, we accept an error tolerance of 1e-3 (0.001). It is not enough if the model gives
nearly the same output, they have to be almost identical. Therefore, you will certainly compare the intermediate
outputs of the 🤗 Transformers version multiple times against the intermediate outputs of the original implementation of
*brand_new_bert* in which case an **efficient** debugging environment of the original repository is absolutely
important. Here is some advice to make your debugging environment as efficient as possible.

- Find the best way of debugging intermediate results. Is the original repository written in PyTorch?


key: _node_content
string_value:
{"id_": "594bc362-1b95-4fb8-b421-79738ccfe2a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f5b4c0b5-98f1-4482-86a7-c6e152a48b07", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "2e8c025d4eebfd604ab599cbaedc37b9d2f3e011e92ce8113bad947ca548e503", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c239810-7002-40f1-a682-f6b67a46acf5", "node_type": "1", "metadata": {}, "hash": "eaff508018d10e9ea1cd7cc466d9d0b30ca83da4f9f8d714ff6c7e3c3a855dde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21410, "end_char_idx": 24711, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since it is normal that the exact same model written in different libraries can give a slightly different output
depending on the library framework, we accept an error tolerance of 1e-3 (0.001). It is not enough if the model gives
nearly the same output, they have to be almost identical. Therefore, you will certainly compare the intermediate
outputs of the 🤗 Transformers version multiple times against the intermediate outputs of the original implementation of
*brand_new_bert* in which case an **efficient** debugging environment of the original repository is absolutely
important. Here is some advice to make your debugging environment as efficient as possible.

- Find the best way of debugging intermediate results. Is the original repository written in PyTorch? Then you should
  probably take the time to write a longer script that decomposes the original model into smaller sub-components to
  retrieve intermediate values. Is the original repository written in Tensorflow 1? Then you might have to rely on
  TensorFlow print operations like [tf.print](https://www.tensorflow.org/api_docs/python/tf/print) to output
  intermediate values. Is the original repository written in Jax? Then make sure that the model is **not jitted** when
  running the forward pass, *e.g.* check-out [this link](https://github.com/google/jax/issues/196).
- Use the smallest pretrained checkpoint you can find. The smaller the checkpoint, the faster your debug cycle
  becomes. It is not efficient if your pretrained model is so big that your forward pass takes more than 10 seconds.
  In case only very large checkpoints are available, it might make more sense to create a dummy model in the new
  environment with randomly initialized weights and save those weights for comparison with the 🤗 Transformers version
  of your model
- Make sure you are using the easiest way of calling a forward pass in the original repository. Ideally, you want to
  find the function in the original repository that **only** calls a single forward pass, *i.e.* that is often called
  `predict`, `evaluate`, `forward` or `__call__`. You don't want to debug a function that calls `forward`
  multiple times, *e.g.* to generate text, like `autoregressive_sample`, `generate`.
- Try to separate the tokenization from the model's *forward* pass. If the original repository shows examples where
  you have to input a string, then try to find out where in the forward call the string input is changed to input ids
  and start from this point. This might mean that you have to possibly write a small script yourself or change the
  original code so that you can directly input the ids instead of an input string.
- Make sure that the model in your debugging setup is **not** in training mode, which often causes the model to yield
  random outputs due to multiple dropout layers in the model. Make sure that the forward pass in your debugging
  environment is **deterministic** so that the dropout layers are not used. Or use *transformers.utils.set_seed*
  if the old and new implementations are in the same framework.

The following section gives you more specific details/tips on how you can do this for *brand_new_bert*.

### 5.-14. Port BrandNewBert to 🤗 Transformers

Next, you can finally start adding new code to 🤗 Transformers.


key: _node_content
string_value:
{"id_": "7c239810-7002-40f1-a682-f6b67a46acf5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "594bc362-1b95-4fb8-b421-79738ccfe2a7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "9ada35ed9444acbc9b74a97bd73c7dae6316f40cb8d39dc6d525b6f8665ce218", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd4afb93-e2ef-45d4-a0bb-9e9321e9a561", "node_type": "1", "metadata": {}, "hash": "e921c7fee3dbba93ca2ac34a7691c11fa78995341a54135017835230b585ecbe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22985, "end_char_idx": 26304, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In case only very large checkpoints are available, it might make more sense to create a dummy model in the new
  environment with randomly initialized weights and save those weights for comparison with the 🤗 Transformers version
  of your model
- Make sure you are using the easiest way of calling a forward pass in the original repository. Ideally, you want to
  find the function in the original repository that **only** calls a single forward pass, *i.e.* that is often called
  `predict`, `evaluate`, `forward` or `__call__`. You don't want to debug a function that calls `forward`
  multiple times, *e.g.* to generate text, like `autoregressive_sample`, `generate`.
- Try to separate the tokenization from the model's *forward* pass. If the original repository shows examples where
  you have to input a string, then try to find out where in the forward call the string input is changed to input ids
  and start from this point. This might mean that you have to possibly write a small script yourself or change the
  original code so that you can directly input the ids instead of an input string.
- Make sure that the model in your debugging setup is **not** in training mode, which often causes the model to yield
  random outputs due to multiple dropout layers in the model. Make sure that the forward pass in your debugging
  environment is **deterministic** so that the dropout layers are not used. Or use *transformers.utils.set_seed*
  if the old and new implementations are in the same framework.

The following section gives you more specific details/tips on how you can do this for *brand_new_bert*.

### 5.-14. Port BrandNewBert to 🤗 Transformers

Next, you can finally start adding new code to 🤗 Transformers. Go into the clone of your 🤗 Transformers' fork:

```bash
cd transformers
```

In the special case that you are adding a model whose architecture exactly matches the model architecture of an
existing model you only have to add a conversion script as described in [this section](#write-a-conversion-script).
In this case, you can just re-use the whole model architecture of the already existing model.

Otherwise, let's start generating a new model. We recommend using the following script to add a model starting from
an existing model:

```bash
transformers-cli add-new-model-like
```

You will be prompted with a questionnaire to fill in the basic information of your model.

**Open a Pull Request on the main huggingface/transformers repo**

Before starting to adapt the automatically generated code, now is the time to open a “Work in progress (WIP)” pull
request, *e.g.* “[WIP] Add *brand_new_bert*”, in 🤗 Transformers so that you and the Hugging Face team can work
side-by-side on integrating the model into 🤗 Transformers.

You should do the following:

1. Create a branch with a descriptive name from your main branch

   ```bash
   git checkout -b add_brand_new_bert
   ```

2. Commit the automatically generated code:

   ```bash
   git add .
   git commit
   ```

3. Fetch and rebase to current main

   ```bash
   git fetch upstream
   git rebase upstream/main
   ```

4. Push the changes to your account using:

   ```bash
   git push -u origin a-descriptive-name-for-my-changes
   ```

5. Once you are satisfied, go to the webpage of your fork on GitHub. Click on “Pull request”.


key: _node_content
string_value:
{"id_": "dd4afb93-e2ef-45d4-a0bb-9e9321e9a561", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c239810-7002-40f1-a682-f6b67a46acf5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "aed4a4fa678cc069f832a5ca2526338f94a8deb9021ac1c10d13f746c6e35a4a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "80afb18d-3d2e-45aa-a61c-461d149c4132", "node_type": "1", "metadata": {}, "hash": "20f149cb0d6b0f8977e675d2a7146ad47e9912ee7cc77612ac04c2e637a54a32", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24712, "end_char_idx": 28028, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Go into the clone of your 🤗 Transformers' fork:

```bash
cd transformers
```

In the special case that you are adding a model whose architecture exactly matches the model architecture of an
existing model you only have to add a conversion script as described in [this section](#write-a-conversion-script).
In this case, you can just re-use the whole model architecture of the already existing model.

Otherwise, let's start generating a new model. We recommend using the following script to add a model starting from
an existing model:

```bash
transformers-cli add-new-model-like
```

You will be prompted with a questionnaire to fill in the basic information of your model.

**Open a Pull Request on the main huggingface/transformers repo**

Before starting to adapt the automatically generated code, now is the time to open a “Work in progress (WIP)” pull
request, *e.g.* “[WIP] Add *brand_new_bert*”, in 🤗 Transformers so that you and the Hugging Face team can work
side-by-side on integrating the model into 🤗 Transformers.

You should do the following:

1. Create a branch with a descriptive name from your main branch

   ```bash
   git checkout -b add_brand_new_bert
   ```

2. Commit the automatically generated code:

   ```bash
   git add .
   git commit
   ```

3. Fetch and rebase to current main

   ```bash
   git fetch upstream
   git rebase upstream/main
   ```

4. Push the changes to your account using:

   ```bash
   git push -u origin a-descriptive-name-for-my-changes
   ```

5. Once you are satisfied, go to the webpage of your fork on GitHub. Click on “Pull request”. Make sure to add the
   GitHub handle of some members of the Hugging Face team as reviewers, so that the Hugging Face team gets notified for
   future changes.

6. Change the PR into a draft by clicking on “Convert to draft” on the right of the GitHub pull request web page.

In the following, whenever you have made some progress, don't forget to commit your work and push it to your account so
that it shows in the pull request. Additionally, you should make sure to update your work with the current main from
time to time by doing:

```bash
git fetch upstream
git merge upstream/main
```

In general, all questions you might have regarding the model or your implementation should be asked in your PR and
discussed/solved in the PR. This way, the Hugging Face team will always be notified when you are committing new code or
if you have a question. It is often very helpful to point the Hugging Face team to your added code so that the Hugging
Face team can efficiently understand your problem or question.

To do so, you can go to the “Files changed” tab where you see all of your changes, go to a line regarding which you
want to ask a question, and click on the “+” symbol to add a comment. Whenever a question or problem has been solved,
you can click on the “Resolve” button of the created comment.

In the same way, the Hugging Face team will open comments when reviewing your code. We recommend asking most questions
on GitHub on your PR. For some very general questions that are not very useful for the public, feel free to ping the
Hugging Face team by Slack or email.

**5. Adapt the generated models code for brand_new_bert**

At first, we will focus only on the model itself and not care about the tokenizer.


key: _node_content
string_value:
{"id_": "80afb18d-3d2e-45aa-a61c-461d149c4132", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd4afb93-e2ef-45d4-a0bb-9e9321e9a561", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "248c8d63a5f70a05125001442207460dd278df6b942fd7743725879f01aa69ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cc86e993-1b99-4b42-b268-36d898a5d378", "node_type": "1", "metadata": {}, "hash": "8e2212f255b0b3d4ec2a1f39de3541255cf8cdbf9dfcde027eba571fa7784bd2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26305, "end_char_idx": 29599, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Make sure to add the
   GitHub handle of some members of the Hugging Face team as reviewers, so that the Hugging Face team gets notified for
   future changes.

6. Change the PR into a draft by clicking on “Convert to draft” on the right of the GitHub pull request web page.

In the following, whenever you have made some progress, don't forget to commit your work and push it to your account so
that it shows in the pull request. Additionally, you should make sure to update your work with the current main from
time to time by doing:

```bash
git fetch upstream
git merge upstream/main
```

In general, all questions you might have regarding the model or your implementation should be asked in your PR and
discussed/solved in the PR. This way, the Hugging Face team will always be notified when you are committing new code or
if you have a question. It is often very helpful to point the Hugging Face team to your added code so that the Hugging
Face team can efficiently understand your problem or question.

To do so, you can go to the “Files changed” tab where you see all of your changes, go to a line regarding which you
want to ask a question, and click on the “+” symbol to add a comment. Whenever a question or problem has been solved,
you can click on the “Resolve” button of the created comment.

In the same way, the Hugging Face team will open comments when reviewing your code. We recommend asking most questions
on GitHub on your PR. For some very general questions that are not very useful for the public, feel free to ping the
Hugging Face team by Slack or email.

**5. Adapt the generated models code for brand_new_bert**

At first, we will focus only on the model itself and not care about the tokenizer. All the relevant code should be
found in the generated files `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` and
`src/transformers/models/brand_new_bert/configuration_brand_new_bert.py`.

Now you can finally start coding :). The generated code in
`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` will either have the same architecture as BERT if
it's an encoder-only model or BART if it's an encoder-decoder model. At this point, you should remind yourself what
you've learned in the beginning about the theoretical aspects of the model: *How is the model different from BERT or
BART?*". Implement those changes which often means changing the *self-attention* layer, the order of the normalization
layer, etc… Again, it is often useful to look at the similar architecture of already existing models in Transformers to
get a better feeling of how your model should be implemented.

**Note** that at this point, you don't have to be very sure that your code is fully correct or clean. Rather, it is
advised to add a first *unclean*, copy-pasted version of the original code to
`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` until you feel like all the necessary code is
added. From our experience, it is much more efficient to quickly add a first version of the required code and
improve/correct the code iteratively with the conversion script as described in the next section. The only thing that
has to work at this point is that you can instantiate the 🤗 Transformers implementation of *brand_new_bert*, *i.e.


key: _node_content
string_value:
{"id_": "cc86e993-1b99-4b42-b268-36d898a5d378", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "80afb18d-3d2e-45aa-a61c-461d149c4132", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "d5829f36c3c4008a64a785966f778ccadf56e69a2c6ff5e6ec209efd67957489", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "33ad78e4-91cd-4b5c-8128-8fac169101e5", "node_type": "1", "metadata": {}, "hash": "db18b816a8bb6f0bb1a4f3592d6a3b6c7ece174defcf84fdd6e00208b8fe390e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27892, "end_char_idx": 31126, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Adapt the generated models code for brand_new_bert**

At first, we will focus only on the model itself and not care about the tokenizer. All the relevant code should be
found in the generated files `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` and
`src/transformers/models/brand_new_bert/configuration_brand_new_bert.py`.

Now you can finally start coding :). The generated code in
`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` will either have the same architecture as BERT if
it's an encoder-only model or BART if it's an encoder-decoder model. At this point, you should remind yourself what
you've learned in the beginning about the theoretical aspects of the model: *How is the model different from BERT or
BART?*". Implement those changes which often means changing the *self-attention* layer, the order of the normalization
layer, etc… Again, it is often useful to look at the similar architecture of already existing models in Transformers to
get a better feeling of how your model should be implemented.

**Note** that at this point, you don't have to be very sure that your code is fully correct or clean. Rather, it is
advised to add a first *unclean*, copy-pasted version of the original code to
`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` until you feel like all the necessary code is
added. From our experience, it is much more efficient to quickly add a first version of the required code and
improve/correct the code iteratively with the conversion script as described in the next section. The only thing that
has to work at this point is that you can instantiate the 🤗 Transformers implementation of *brand_new_bert*, *i.e.* the
following command should work:

```python
from transformers import BrandNewBertModel, BrandNewBertConfig

model = BrandNewBertModel(BrandNewBertConfig())
```

The above command will create a model according to the default parameters as defined in `BrandNewBertConfig()` with
random weights, thus making sure that the `init()` methods of all components works.

Note that all random initialization should happen in the `_init_weights` method of your `BrandnewBertPreTrainedModel`
class. It should initialize all leaf modules depending on the variables of the config. Here is an example with the
BERT `_init_weights` method:

```py
def _init_weights(self, module):
    """Initialize the weights"""
    if isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()
    elif isinstance(module, nn.Embedding):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.padding_idx is not None:
            module.weight.data[module.padding_idx].zero_()
    elif isinstance(module, nn.LayerNorm):
        module.bias.data.zero_()
        module.weight.data.fill_(1.0)
```

You can have some more custom schemes if you need a special initialization for some modules. For instance, in
`Wav2Vec2ForPreTraining`, the last two linear layers need to have the initialization of the regular PyTorch `nn.Linear`
but all the other ones should use an initialization as above.


key: _node_content
string_value:
{"id_": "33ad78e4-91cd-4b5c-8128-8fac169101e5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cc86e993-1b99-4b42-b268-36d898a5d378", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "ae0114515928a2e185b6a6b2ff16e1a6e6a4449445949b82c72b5cfd86b5c0c4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6d82b6a4-d526-478c-95c1-18300c697686", "node_type": "1", "metadata": {}, "hash": "6b72d901339824be5511fecce72586feb08fa401f616b29675add2a5ded0bf0e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29465, "end_char_idx": 32734, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The only thing that
has to work at this point is that you can instantiate the 🤗 Transformers implementation of *brand_new_bert*, *i.e.* the
following command should work:

```python
from transformers import BrandNewBertModel, BrandNewBertConfig

model = BrandNewBertModel(BrandNewBertConfig())
```

The above command will create a model according to the default parameters as defined in `BrandNewBertConfig()` with
random weights, thus making sure that the `init()` methods of all components works.

Note that all random initialization should happen in the `_init_weights` method of your `BrandnewBertPreTrainedModel`
class. It should initialize all leaf modules depending on the variables of the config. Here is an example with the
BERT `_init_weights` method:

```py
def _init_weights(self, module):
    """Initialize the weights"""
    if isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()
    elif isinstance(module, nn.Embedding):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.padding_idx is not None:
            module.weight.data[module.padding_idx].zero_()
    elif isinstance(module, nn.LayerNorm):
        module.bias.data.zero_()
        module.weight.data.fill_(1.0)
```

You can have some more custom schemes if you need a special initialization for some modules. For instance, in
`Wav2Vec2ForPreTraining`, the last two linear layers need to have the initialization of the regular PyTorch `nn.Linear`
but all the other ones should use an initialization as above. This is coded like this:

```py
def _init_weights(self, module):
    """Initialize the weights"""
    if isinstance(module, Wav2Vec2ForPreTraining):
        module.project_hid.reset_parameters()
        module.project_q.reset_parameters()
        module.project_hid._is_hf_initialized = True
        module.project_q._is_hf_initialized = True
    elif isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()
```

The `_is_hf_initialized` flag is internally used to make sure we only initialize a submodule once. By setting it to
`True` for `module.project_q` and `module.project_hid`, we make sure the custom initialization we did is not overridden later on,
the `_init_weights` function won't be applied to them.

**6. Write a conversion script**

Next, you should write a conversion script that lets you convert the checkpoint you used to debug *brand_new_bert* in
the original repository to a checkpoint compatible with your just created 🤗 Transformers implementation of
*brand_new_bert*. It is not advised to write the conversion script from scratch, but rather to look through already
existing conversion scripts in 🤗 Transformers for one that has been used to convert a similar model that was written in
the same framework as *brand_new_bert*. Usually, it is enough to copy an already existing conversion script and
slightly adapt it for your use case. Don't hesitate to ask the Hugging Face team to point you to a similar already
existing conversion script for your model.


key: _node_content
string_value:
{"id_": "6d82b6a4-d526-478c-95c1-18300c697686", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "33ad78e4-91cd-4b5c-8128-8fac169101e5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "51f094ff6ee5d474b9b26600a36717f2b47f9f47bf2f756cd54aeeea3652f962", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "836f47a5-e9a8-4748-81df-e9ba4c761a48", "node_type": "1", "metadata": {}, "hash": "2ccc8f19db1200e2188c86c7db5e40ba74075a88a1a2956a5fae250f51c6e58a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30928, "end_char_idx": 33917, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For instance, in
`Wav2Vec2ForPreTraining`, the last two linear layers need to have the initialization of the regular PyTorch `nn.Linear`
but all the other ones should use an initialization as above. This is coded like this:

```py
def _init_weights(self, module):
    """Initialize the weights"""
    if isinstance(module, Wav2Vec2ForPreTraining):
        module.project_hid.reset_parameters()
        module.project_q.reset_parameters()
        module.project_hid._is_hf_initialized = True
        module.project_q._is_hf_initialized = True
    elif isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()
```

The `_is_hf_initialized` flag is internally used to make sure we only initialize a submodule once. By setting it to
`True` for `module.project_q` and `module.project_hid`, we make sure the custom initialization we did is not overridden later on,
the `_init_weights` function won't be applied to them.

**6. Write a conversion script**

Next, you should write a conversion script that lets you convert the checkpoint you used to debug *brand_new_bert* in
the original repository to a checkpoint compatible with your just created 🤗 Transformers implementation of
*brand_new_bert*. It is not advised to write the conversion script from scratch, but rather to look through already
existing conversion scripts in 🤗 Transformers for one that has been used to convert a similar model that was written in
the same framework as *brand_new_bert*. Usually, it is enough to copy an already existing conversion script and
slightly adapt it for your use case. Don't hesitate to ask the Hugging Face team to point you to a similar already
existing conversion script for your model.

- If you are porting a model from TensorFlow to PyTorch, a good starting point might be BERT's conversion script [here](https://github.com/huggingface/transformers/blob/7acfa95afb8194f8f9c1f4d2c6028224dbed35a2/src/transformers/models/bert/modeling_bert.py#L91)
- If you are porting a model from PyTorch to PyTorch, a good starting point might be BART's conversion script [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/convert_bart_original_pytorch_checkpoint_to_pytorch.py)

In the following, we'll quickly explain how PyTorch models store layer weights and define layer names. In PyTorch, the
name of a layer is defined by the name of the class attribute you give the layer. Let's define a dummy model in
PyTorch, called `SimpleModel` as follows:

```python
from torch import nn


class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.dense = nn.Linear(10, 10)
        self.intermediate = nn.Linear(10, 10)
        self.layer_norm = nn.LayerNorm(10)
```

Now we can create an instance of this model definition which will fill all weights: `dense`, `intermediate`,
`layer_norm` with random weights.


key: _node_content
string_value:
{"id_": "836f47a5-e9a8-4748-81df-e9ba4c761a48", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6d82b6a4-d526-478c-95c1-18300c697686", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "50658d17d1e104ffb45ae39b0bdd7bd16bcfb99a8be36400a1e6f5a05a8ab61c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e44d8db-ad92-46e8-bc48-445875cbb0c0", "node_type": "1", "metadata": {}, "hash": "dcc459908fc9931a611151b70f769b6dc407c463e9f82a6a53b2d35720959c46", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32505, "end_char_idx": 34367, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Usually, it is enough to copy an already existing conversion script and
slightly adapt it for your use case. Don't hesitate to ask the Hugging Face team to point you to a similar already
existing conversion script for your model.

- If you are porting a model from TensorFlow to PyTorch, a good starting point might be BERT's conversion script [here](https://github.com/huggingface/transformers/blob/7acfa95afb8194f8f9c1f4d2c6028224dbed35a2/src/transformers/models/bert/modeling_bert.py#L91)
- If you are porting a model from PyTorch to PyTorch, a good starting point might be BART's conversion script [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/convert_bart_original_pytorch_checkpoint_to_pytorch.py)

In the following, we'll quickly explain how PyTorch models store layer weights and define layer names. In PyTorch, the
name of a layer is defined by the name of the class attribute you give the layer. Let's define a dummy model in
PyTorch, called `SimpleModel` as follows:

```python
from torch import nn


class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.dense = nn.Linear(10, 10)
        self.intermediate = nn.Linear(10, 10)
        self.layer_norm = nn.LayerNorm(10)
```

Now we can create an instance of this model definition which will fill all weights: `dense`, `intermediate`,
`layer_norm` with random weights. We can print the model to see its architecture

```python
model = SimpleModel()

print(model)
```

This will print out the following:

```
SimpleModel(
  (dense): Linear(in_features=10, out_features=10, bias=True)
  (intermediate): Linear(in_features=10, out_features=10, bias=True)
  (layer_norm): LayerNorm((10,), eps=1e-05, elementwise_affine=True)
)
```

We can see that the layer names are defined by the name of the class attribute in PyTorch.


key: _node_content
string_value:
{"id_": "7e44d8db-ad92-46e8-bc48-445875cbb0c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "836f47a5-e9a8-4748-81df-e9ba4c761a48", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "3beec97ae0ec67f8e1ebedb2404ffc24efe524b51848143cc43ca0ecd5f9907a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "276b2f38-97ee-49d5-9488-b974f3c867c5", "node_type": "1", "metadata": {}, "hash": "85865ed2b2bb37e4c963177732ba9640fd0c81396e8aa0c9a6463ce10c00cbb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33357, "end_char_idx": 35618, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In PyTorch, the
name of a layer is defined by the name of the class attribute you give the layer. Let's define a dummy model in
PyTorch, called `SimpleModel` as follows:

```python
from torch import nn


class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.dense = nn.Linear(10, 10)
        self.intermediate = nn.Linear(10, 10)
        self.layer_norm = nn.LayerNorm(10)
```

Now we can create an instance of this model definition which will fill all weights: `dense`, `intermediate`,
`layer_norm` with random weights. We can print the model to see its architecture

```python
model = SimpleModel()

print(model)
```

This will print out the following:

```
SimpleModel(
  (dense): Linear(in_features=10, out_features=10, bias=True)
  (intermediate): Linear(in_features=10, out_features=10, bias=True)
  (layer_norm): LayerNorm((10,), eps=1e-05, elementwise_affine=True)
)
```

We can see that the layer names are defined by the name of the class attribute in PyTorch. You can print out the weight
values of a specific layer:

```python
print(model.dense.weight.data)
```

to see that the weights were randomly initialized

```
tensor([[-0.0818,  0.2207, -0.0749, -0.0030,  0.0045, -0.1569, -0.1598,  0.0212,
         -0.2077,  0.2157],
        [ 0.1044,  0.0201,  0.0990,  0.2482,  0.3116,  0.2509,  0.2866, -0.2190,
          0.2166, -0.0212],
        [-0.2000,  0.1107, -0.1999, -0.3119,  0.1559,  0.0993,  0.1776, -0.1950,
         -0.1023, -0.0447],
        [-0.0888, -0.1092,  0.2281,  0.0336,  0.1817, -0.0115,  0.2096,  0.1415,
         -0.1876, -0.2467],
        [ 0.2208, -0.2352, -0.1426, -0.2636, -0.2889, -0.2061, -0.2849, -0.0465,
          0.2577,  0.0402],
        [ 0.1502,  0.2465,  0.2566,  0.0693,  0.2352, -0.0530,  0.1859, -0.0604,
          0.2132,  0.1680],
        [ 0.1733, -0.2407, -0.1721,  0.1484,  0.0358, -0.0633, -0.0721, -0.0090,
          0.2707, -0.2509],
        [-0.1173,  0.1561,  0.2945,  0.0595, -0.1996,  0.2988, -0.0802,  0.0407,
          0.1829, -0.1568],
        [-0.1164, -0.2228, -0.0403,  0.0428,  0.1339,  0.0047,  0.1967,  0.2923,
          0.0333, -0.0536],
        [-0.1492, -0.1616,  0.1057,  0.1950, -0.2807, -0.2710, -0.1586,  0.0739,
          0.2220,  0.2358]]).


key: _node_content
string_value:
{"id_": "276b2f38-97ee-49d5-9488-b974f3c867c5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7e44d8db-ad92-46e8-bc48-445875cbb0c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "1653a302ce09457180eb3ee3d403a09912e2c6447e20111723d5a1e8d3c00884", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e6c0a3c7-10d0-4c91-a914-be3e0ecfd55a", "node_type": "1", "metadata": {}, "hash": "1eb8bdbc68f31d016c44971931cb0e0b77d67b3197d8f4d9164c75635f50ff1e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35619, "end_char_idx": 38901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

In the conversion script, you should fill those randomly initialized weights with the exact weights of the
corresponding layer in the checkpoint. *E.g.*

```python
# retrieve matching layer weights, e.g. by
# recursive algorithm
layer_name = "dense"
pretrained_weight = array_of_dense_layer

model_pointer = getattr(model, "dense")

model_pointer.weight.data = torch.from_numpy(pretrained_weight)
```

While doing so, you must verify that each randomly initialized weight of your PyTorch model and its corresponding
pretrained checkpoint weight exactly match in both **shape and name**. To do so, it is **necessary** to add assert
statements for the shape and print out the names of the checkpoints weights. E.g. you should add statements like:

```python
assert (
    model_pointer.weight.shape == pretrained_weight.shape
), f"Pointer shape of random weight {model_pointer.shape} and array shape of checkpoint weight {pretrained_weight.shape} mismatched"
```

Besides, you should also print out the names of both weights to make sure they match, *e.g.*

```python
logger.info(f"Initialize PyTorch weight {layer_name} from {pretrained_weight.name}")
```

If either the shape or the name doesn't match, you probably assigned the wrong checkpoint weight to a randomly
initialized layer of the 🤗 Transformers implementation.

An incorrect shape is most likely due to an incorrect setting of the config parameters in `BrandNewBertConfig()` that
do not exactly match those that were used for the checkpoint you want to convert. However, it could also be that
PyTorch's implementation of a layer requires the weight to be transposed beforehand.

Finally, you should also check that **all** required weights are initialized and print out all checkpoint weights that
were not used for initialization to make sure the model is correctly converted. It is completely normal, that the
conversion trials fail with either a wrong shape statement or a wrong name assignment. This is most likely because either
you used incorrect parameters in `BrandNewBertConfig()`, have a wrong architecture in the 🤗 Transformers
implementation, you have a bug in the `init()` functions of one of the components of the 🤗 Transformers
implementation or you need to transpose one of the checkpoint weights.

This step should be iterated with the previous step until all weights of the checkpoint are correctly loaded in the
Transformers model. Having correctly loaded the checkpoint into the 🤗 Transformers implementation, you can then save
the model under a folder of your choice `/path/to/converted/checkpoint/folder` that should then contain both a
`pytorch_model.bin` file and a `config.json` file:

```python
model.save_pretrained("/path/to/converted/checkpoint/folder")
```

**7. Implement the forward pass**

Having managed to correctly load the pretrained weights into the 🤗 Transformers implementation, you should now make
sure that the forward pass is correctly implemented. In [Get familiar with the original repository](#3-4-run-a-pretrained-checkpoint-using-the-original-repository), you have already created a script that runs a forward
pass of the model using the original repository. Now you should write an analogous script using the 🤗 Transformers
implementation instead of the original one.


key: _node_content
string_value:
{"id_": "e6c0a3c7-10d0-4c91-a914-be3e0ecfd55a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "276b2f38-97ee-49d5-9488-b974f3c867c5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "b2c0164edb9c4a63a308faab12e4616eebd84072d12d80a9b83c10f00983a1c0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7afbc7a8-b1d8-4622-8178-ffb2d2e6dcd5", "node_type": "1", "metadata": {}, "hash": "5334815d10b82c623f1d6822a7028d94bfda04723221d7bd352af05b898a215a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37147, "end_char_idx": 40667, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, it could also be that
PyTorch's implementation of a layer requires the weight to be transposed beforehand.

Finally, you should also check that **all** required weights are initialized and print out all checkpoint weights that
were not used for initialization to make sure the model is correctly converted. It is completely normal, that the
conversion trials fail with either a wrong shape statement or a wrong name assignment. This is most likely because either
you used incorrect parameters in `BrandNewBertConfig()`, have a wrong architecture in the 🤗 Transformers
implementation, you have a bug in the `init()` functions of one of the components of the 🤗 Transformers
implementation or you need to transpose one of the checkpoint weights.

This step should be iterated with the previous step until all weights of the checkpoint are correctly loaded in the
Transformers model. Having correctly loaded the checkpoint into the 🤗 Transformers implementation, you can then save
the model under a folder of your choice `/path/to/converted/checkpoint/folder` that should then contain both a
`pytorch_model.bin` file and a `config.json` file:

```python
model.save_pretrained("/path/to/converted/checkpoint/folder")
```

**7. Implement the forward pass**

Having managed to correctly load the pretrained weights into the 🤗 Transformers implementation, you should now make
sure that the forward pass is correctly implemented. In [Get familiar with the original repository](#3-4-run-a-pretrained-checkpoint-using-the-original-repository), you have already created a script that runs a forward
pass of the model using the original repository. Now you should write an analogous script using the 🤗 Transformers
implementation instead of the original one. It should look as follows:

```python
model = BrandNewBertModel.from_pretrained("/path/to/converted/checkpoint/folder")
input_ids = [0, 4, 4, 3, 2, 4, 1, 7, 19]
output = model(input_ids).last_hidden_states
```

It is very likely that the 🤗 Transformers implementation and the original model implementation don't give the exact
same output the very first time or that the forward pass throws an error. Don't be disappointed - it's expected! First,
you should make sure that the forward pass doesn't throw any errors. It often happens that the wrong dimensions are
used leading to a *Dimensionality mismatch* error or that the wrong data type object is used, *e.g.* `torch.long`
instead of `torch.float32`. Don't hesitate to ask the Hugging Face team for help, if you don't manage to solve
certain errors.

The final part to make sure the 🤗 Transformers implementation works correctly is to ensure that the outputs are
equivalent to a precision of `1e-3`. First, you should ensure that the output shapes are identical, *i.e.*
`outputs.shape` should yield the same value for the script of the 🤗 Transformers implementation and the original
implementation. Next, you should make sure that the output values are identical as well. This one of the most difficult
parts of adding a new model. Common mistakes why the outputs are not identical are:

- Some layers were not added, *i.e.* an *activation* layer was not added, or the residual connection was forgotten
- The word embedding matrix was not tied
- The wrong positional embeddings are used because the original implementation uses on offset
- Dropout is applied during the forward pass. To fix this make sure *model.training is False* and that no dropout
  layer is falsely activated during the forward pass, *i.e.


key: _node_content
string_value:
{"id_": "7afbc7a8-b1d8-4622-8178-ffb2d2e6dcd5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e6c0a3c7-10d0-4c91-a914-be3e0ecfd55a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "af050468cd3d7414bc1b8766e57295785252abf8d44d229e8e1d025adf7e32b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "72bf632b-4866-43fb-90b4-30c2c50c5922", "node_type": "1", "metadata": {}, "hash": "9b16a2c9b53d32006f57f6fadb5eb275d23da687f1e3cdb18e4db61108186782", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39303, "end_char_idx": 42701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


