# embedding_metadata

key: chroma:document
string_value:
"},
]
print(pipe(messages, max_new_tokens=128)[0]['generated_text'][-1])  # Print the assistant's response
```

```text
{'role': 'assistant', 'content': "Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all."}
```

The pipeline will take care of all the details of tokenization and calling `apply_chat_template` for you -
once the model has a chat template, all you need to do is initialize the pipeline and pass it the list of messages!

## What are "generation prompts"?

You may have noticed that the `apply_chat_template` method has an `add_generation_prompt` argument. This argument tells
the template to add tokens that indicate the start of a bot response. For example, consider the following chat:

```python
messages = [
    {"role": "user", "content": "Hi there!"},
    {"role": "assistant", "content": "Nice to meet you!"},
    {"role": "user", "content": "Can I ask a question?"}
]
```

Here's what this will look like without a generation prompt, using the ChatML template we saw in the Zephyr example:

```python
tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
"""<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
"""
```

And here's what it looks like **with** a generation prompt:

```python
tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
"""<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant
"""
```

Note that this time, we've added the tokens that indicate the start of a bot response. This ensures that when the model
generates text it will write a bot response instead of doing something unexpected, like continuing the user's 
message. Remember, chat models are still just language models - they're trained to continue text, and chat is just a 
special kind of text to them! You need to guide them with appropriate control tokens, so they know what they're 
supposed to be doing.

Not all models require generation prompts. Some models, like BlenderBot and LLaMA, don't have any
special tokens before bot responses. In these cases, the `add_generation_prompt` argument will have no effect. The exact
effect that `add_generation_prompt` has will depend on the template being used.

## Can I use chat templates in training?

Yes! This is a good way to ensure that the chat template matches the tokens the model sees during training.
We recommend that you apply the chat template as a preprocessing step for your dataset. After this, you
can simply continue like any other language model training task. When training, you should usually set 
`add_generation_prompt=False`, because the added tokens to prompt an assistant response will not be helpful during 
training.


key: _node_content
string_value:
{"id_": "bf38b199-892d-416b-af40-392b0fe419ec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3e27a075-f85f-461b-9e20-b17b7a82bd5f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "342a1368156d1a713d49a5ac3b0e35fe72e050bf9af9859ee0dd8163fe4ad726", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8d7e8eeb-120b-48b8-a252-4e0572c9dc4c", "node_type": "1", "metadata": {}, "hash": "e3cf0e2636c48f392bcfe532b3b38d246be63622158f9084551ace33e7749357", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7637, "end_char_idx": 11048, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This ensures that when the model
generates text it will write a bot response instead of doing something unexpected, like continuing the user's 
message. Remember, chat models are still just language models - they're trained to continue text, and chat is just a 
special kind of text to them! You need to guide them with appropriate control tokens, so they know what they're 
supposed to be doing.

Not all models require generation prompts. Some models, like BlenderBot and LLaMA, don't have any
special tokens before bot responses. In these cases, the `add_generation_prompt` argument will have no effect. The exact
effect that `add_generation_prompt` has will depend on the template being used.

## Can I use chat templates in training?

Yes! This is a good way to ensure that the chat template matches the tokens the model sees during training.
We recommend that you apply the chat template as a preprocessing step for your dataset. After this, you
can simply continue like any other language model training task. When training, you should usually set 
`add_generation_prompt=False`, because the added tokens to prompt an assistant response will not be helpful during 
training. Let's see an example:

```python
from transformers import AutoTokenizer
from datasets import Dataset

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")

chat1 = [
    {"role": "user", "content": "Which is bigger, the moon or the sun?"},
    {"role": "assistant", "content": "The sun."}
]
chat2 = [
    {"role": "user", "content": "Which is bigger, a virus or a bacterium?"},
    {"role": "assistant", "content": "A bacterium."}
]

dataset = Dataset.from_dict({"chat": [chat1, chat2]})
dataset = dataset.map(lambda x: {"formatted_chat": tokenizer.apply_chat_template(x["chat"], tokenize=False, add_generation_prompt=False)})
print(dataset['formatted_chat'][0])
```
And we get:
```text
<|user|>
Which is bigger, the moon or the sun?</s>
<|assistant|>
The sun.</s>
```

From here, just continue training like you would with a standard language modelling task, using the `formatted_chat` column.

<Tip>

By default, some tokenizers add special tokens like `<bos>` and `<eos>` to text they tokenize. Chat templates should 
already include all the special tokens they need, and so additional special tokens will often be incorrect or 
duplicated, which will hurt model performance.

Therefore, if you format text with `apply_chat_template(tokenize=False)`, you should set the argument
`add_special_tokens=False` when you tokenize that text later. If you use `apply_chat_template(tokenize=True)`, you don't need to worry about this!

</Tip>

## Advanced: Extra inputs to chat templates

The only argument that `apply_chat_template` requires is `messages`. However, you can pass any keyword
argument to `apply_chat_template` and it will be accessible inside the template. This gives you a lot of freedom to use
chat templates for many things. There are no restrictions on the names or the format of these arguments - you can pass
strings, lists, dicts or whatever else you want. 

That said, there are some common use-cases for these extra arguments,
such as passing tools for function calling, or documents for retrieval-augmented generation. In these common cases,
we have some opinionated recommendations about what the names and formats of these arguments should be, which are
described in the sections below.


key: _node_content
string_value:
{"id_": "8d7e8eeb-120b-48b8-a252-4e0572c9dc4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bf38b199-892d-416b-af40-392b0fe419ec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "415b277840a4f58a67885461492a40c2ee721489669ecd8dcd07f797bbb3c4aa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b14e7a7c-b168-4053-8aa8-1e2082e1a7e8", "node_type": "1", "metadata": {}, "hash": "ca6d1e2c0e3f1950bb0fd0fc6f742ffb707de67dd5fac93e23d02bca5a12a393", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9271, "end_char_idx": 12737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
]

dataset = Dataset.from_dict({"chat": [chat1, chat2]})
dataset = dataset.map(lambda x: {"formatted_chat": tokenizer.apply_chat_template(x["chat"], tokenize=False, add_generation_prompt=False)})
print(dataset['formatted_chat'][0])
```
And we get:
```text
<|user|>
Which is bigger, the moon or the sun?</s>
<|assistant|>
The sun.</s>
```

From here, just continue training like you would with a standard language modelling task, using the `formatted_chat` column.

<Tip>

By default, some tokenizers add special tokens like `<bos>` and `<eos>` to text they tokenize. Chat templates should 
already include all the special tokens they need, and so additional special tokens will often be incorrect or 
duplicated, which will hurt model performance.

Therefore, if you format text with `apply_chat_template(tokenize=False)`, you should set the argument
`add_special_tokens=False` when you tokenize that text later. If you use `apply_chat_template(tokenize=True)`, you don't need to worry about this!

</Tip>

## Advanced: Extra inputs to chat templates

The only argument that `apply_chat_template` requires is `messages`. However, you can pass any keyword
argument to `apply_chat_template` and it will be accessible inside the template. This gives you a lot of freedom to use
chat templates for many things. There are no restrictions on the names or the format of these arguments - you can pass
strings, lists, dicts or whatever else you want. 

That said, there are some common use-cases for these extra arguments,
such as passing tools for function calling, or documents for retrieval-augmented generation. In these common cases,
we have some opinionated recommendations about what the names and formats of these arguments should be, which are
described in the sections below. We encourage model authors to make their chat templates compatible with this format,
to make it easy to transfer tool-calling code between models.

## Advanced: Tool use / function calling

"Tool use" LLMs can choose to call functions as external tools before generating an answer. When passing tools
to a tool-use model, you can simply pass a list of functions to the `tools` argument:

```python
import datetime

def current_time():
    """Get the current local time as a string."""
    return str(datetime.now())

def multiply(a: float, b: float):
    """
    A function that multiplies two numbers
    
    Args:
        a: The first number to multiply
        b: The second number to multiply
    """
    return a * b

tools = [current_time, multiply]

model_input = tokenizer.apply_chat_template(
    messages,
    tools=tools
)
```

In order for this to work correctly, you should write your functions in the format above, so that they can be parsed
correctly as tools. Specifically, you should follow these rules:

- The function should have a descriptive name
- Every argument must have a type hint
- The function must have a docstring in the standard Google style (in other words, an initial function description  
  followed by an `Args:` block that describes the arguments, unless the function does not have any arguments. 
- Do not include types in the `Args:` block. In other words, write `a: The first number to multiply`, not
  `a (int): The first number to multiply`. Type hints should go in the function header instead.
- The function can have a return type and a `Returns:` block in the docstring. However, these are optional
  because most tool-use models ignore them.


key: _node_content
string_value:
{"id_": "b14e7a7c-b168-4053-8aa8-1e2082e1a7e8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8d7e8eeb-120b-48b8-a252-4e0572c9dc4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "b25a8df51846d7d7d90b12a7dafa6fa44b966afb561f785642ff6d76b87334a7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "752cd2d9-f8ba-42da-a5d9-795c02c5efc6", "node_type": "1", "metadata": {}, "hash": "8cdfec37eeee847545b715aaf2adc2aa2d98221039924d9c8aa09c3ca61ad579", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11049, "end_char_idx": 14208, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We encourage model authors to make their chat templates compatible with this format,
to make it easy to transfer tool-calling code between models.

## Advanced: Tool use / function calling

"Tool use" LLMs can choose to call functions as external tools before generating an answer. When passing tools
to a tool-use model, you can simply pass a list of functions to the `tools` argument:

```python
import datetime

def current_time():
    """Get the current local time as a string."""
    return str(datetime.now())

def multiply(a: float, b: float):
    """
    A function that multiplies two numbers
    
    Args:
        a: The first number to multiply
        b: The second number to multiply
    """
    return a * b

tools = [current_time, multiply]

model_input = tokenizer.apply_chat_template(
    messages,
    tools=tools
)
```

In order for this to work correctly, you should write your functions in the format above, so that they can be parsed
correctly as tools. Specifically, you should follow these rules:

- The function should have a descriptive name
- Every argument must have a type hint
- The function must have a docstring in the standard Google style (in other words, an initial function description  
  followed by an `Args:` block that describes the arguments, unless the function does not have any arguments. 
- Do not include types in the `Args:` block. In other words, write `a: The first number to multiply`, not
  `a (int): The first number to multiply`. Type hints should go in the function header instead.
- The function can have a return type and a `Returns:` block in the docstring. However, these are optional
  because most tool-use models ignore them.

### Passing tool results to the model

The sample code above is enough to list the available tools for your model, but what happens if it wants to actually use
one? If that happens, you should:

1. Parse the model's output to get the tool name(s) and arguments.
2. Add the model's tool call(s) to the conversation.
3. Call the corresponding function(s) with those arguments.
4. Add the result(s) to the conversation

### A complete tool use example

Let's walk through a tool use example, step by step. For this example, we will use an 8B `Hermes-2-Pro` model,
as it is one of the highest-performing tool-use models in its size category at the time of writing. If you have the
memory, you can consider using a larger model instead like [Command-R](https://huggingface.co/CohereForAI/c4ai-command-r-v01)
or [Mixtral-8x22B](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), both of which also support tool use
and offer even stronger performance.

First, let's load our model and tokenizer:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "NousResearch/Hermes-2-Pro-Llama-3-8B"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto")
```

Next, let's define a list of tools:

```python
def get_current_temperature(location: str, unit: str) -> float:
    """
    Get the current temperature at a location.


key: _node_content
string_value:
{"id_": "752cd2d9-f8ba-42da-a5d9-795c02c5efc6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b14e7a7c-b168-4053-8aa8-1e2082e1a7e8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "96fd75ffd453a7b8fd8295e84313ff78c8bf1ac62a26143c2e4eb8d15fdeb3c9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7bbbc7ae-c29e-477e-a96d-e2ccacbd5828", "node_type": "1", "metadata": {}, "hash": "aaba5a100fd20d4fa6d55c694cf25cce967254ff1a2dc0eff1f1e0e6f2a7fc72", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12666, "end_char_idx": 15355, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, these are optional
  because most tool-use models ignore them.

### Passing tool results to the model

The sample code above is enough to list the available tools for your model, but what happens if it wants to actually use
one? If that happens, you should:

1. Parse the model's output to get the tool name(s) and arguments.
2. Add the model's tool call(s) to the conversation.
3. Call the corresponding function(s) with those arguments.
4. Add the result(s) to the conversation

### A complete tool use example

Let's walk through a tool use example, step by step. For this example, we will use an 8B `Hermes-2-Pro` model,
as it is one of the highest-performing tool-use models in its size category at the time of writing. If you have the
memory, you can consider using a larger model instead like [Command-R](https://huggingface.co/CohereForAI/c4ai-command-r-v01)
or [Mixtral-8x22B](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), both of which also support tool use
and offer even stronger performance.

First, let's load our model and tokenizer:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "NousResearch/Hermes-2-Pro-Llama-3-8B"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto")
```

Next, let's define a list of tools:

```python
def get_current_temperature(location: str, unit: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
        unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
    Returns:
        The current temperature at the specified location in the specified units, as a float.
    """
    return 22.  # A real function should probably actually get the temperature!

def get_current_wind_speed(location: str) -> float:
    """
    Get the current wind speed in km/h at a given location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current wind speed at the given location in km/h, as a float.
    """
    return 6.  # A real function should probably actually get the wind speed!

tools = [get_current_temperature, get_current_wind_speed]
```

Now, let's set up a conversation for our bot:

```python
messages = [
  {"role": "system", "content": "You are a bot that responds to weather queries. You should reply with the unit used in the queried location."},
  {"role": "user", "content": "Hey, what's the temperature in Paris right now?"}


key: _node_content
string_value:
{"id_": "7bbbc7ae-c29e-477e-a96d-e2ccacbd5828", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "752cd2d9-f8ba-42da-a5d9-795c02c5efc6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "37267e72be107fa5e745be6e93f389df492105d2c51031965685a96949d569f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4ff4d19-54e2-4a5b-9a73-8850dee1aceb", "node_type": "1", "metadata": {}, "hash": "3c4c303ebddb5b95fa9187dbf79e3e1c80c797e6aa762c34941d613f0244fe21", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13698, "end_char_idx": 16754, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
First, let's load our model and tokenizer:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "NousResearch/Hermes-2-Pro-Llama-3-8B"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto")
```

Next, let's define a list of tools:

```python
def get_current_temperature(location: str, unit: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
        unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
    Returns:
        The current temperature at the specified location in the specified units, as a float.
    """
    return 22.  # A real function should probably actually get the temperature!

def get_current_wind_speed(location: str) -> float:
    """
    Get the current wind speed in km/h at a given location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current wind speed at the given location in km/h, as a float.
    """
    return 6.  # A real function should probably actually get the wind speed!

tools = [get_current_temperature, get_current_wind_speed]
```

Now, let's set up a conversation for our bot:

```python
messages = [
  {"role": "system", "content": "You are a bot that responds to weather queries. You should reply with the unit used in the queried location."},
  {"role": "user", "content": "Hey, what's the temperature in Paris right now?"}
]
```

Now, let's apply the chat template and generate a response:

```python
inputs = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_dict=True, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0][len(inputs["input_ids"][0]):]))
```

And we get:

```text
<tool_call>
{"arguments": {"location": "Paris, France", "unit": "celsius"}, "name": "get_current_temperature"}
</tool_call><|im_end|>
```

The model has called the function with valid arguments, in the format requested by the function docstring. It has
inferred that we're most likely referring to the Paris in France, and it remembered that, as the home of SI units,
the temperature in France should certainly be displayed in Celsius.

<Tip>

The output format above is specific to the `Hermes-2-Pro` model we're using in this example. Other models may emit different
tool call formats, and you may need to do some manual parsing at this step. For example, `Llama-3.1` models will emit
slightly different JSON, with `parameters` instead of `arguments`. Regardless of the format the model outputs, you 
should add the tool call to the conversation in the format below, with `tool_calls`, `function` and `arguments` keys. 

</Tip>

Next, let's append the model's tool call to the conversation.


key: _node_content
string_value:
{"id_": "c4ff4d19-54e2-4a5b-9a73-8850dee1aceb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7bbbc7ae-c29e-477e-a96d-e2ccacbd5828", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "cf2b652af92c710e1bf292775fdba6002113f1c39e69f9c32eb6fa10be74ee8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03eec2f3-7071-4178-abc2-dcb07ac9a146", "node_type": "1", "metadata": {}, "hash": "aaf707624247f6785cd6fce2f99f03f1c9d1758a1d716aa389c0bdd0ced4e7d4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15211, "end_char_idx": 17705, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You should reply with the unit used in the queried location."},
  {"role": "user", "content": "Hey, what's the temperature in Paris right now?"}
]
```

Now, let's apply the chat template and generate a response:

```python
inputs = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_dict=True, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0][len(inputs["input_ids"][0]):]))
```

And we get:

```text
<tool_call>
{"arguments": {"location": "Paris, France", "unit": "celsius"}, "name": "get_current_temperature"}
</tool_call><|im_end|>
```

The model has called the function with valid arguments, in the format requested by the function docstring. It has
inferred that we're most likely referring to the Paris in France, and it remembered that, as the home of SI units,
the temperature in France should certainly be displayed in Celsius.

<Tip>

The output format above is specific to the `Hermes-2-Pro` model we're using in this example. Other models may emit different
tool call formats, and you may need to do some manual parsing at this step. For example, `Llama-3.1` models will emit
slightly different JSON, with `parameters` instead of `arguments`. Regardless of the format the model outputs, you 
should add the tool call to the conversation in the format below, with `tool_calls`, `function` and `arguments` keys. 

</Tip>

Next, let's append the model's tool call to the conversation.

```python
tool_call = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
messages.append({"role": "assistant", "tool_calls": [{"type": "function", "function": tool_call}]})
```


Now that we've added the tool call to the conversation, we can call the function and append the result to the
conversation. Since we're just using a dummy function for this example that always returns 22.0, we can just append 
that result directly.

```python
messages.append({"role": "tool", "name": "get_current_temperature", "content": "22.0"})
```

<Tip>

Some model architectures, notably Mistral/Mixtral, also require a `tool_call_id` here, which should be
9 randomly-generated alphanumeric characters, and assigned to the `id` key of the tool call
dictionary. The same key should also be assigned to the `tool_call_id` key of the tool response dictionary below, so 
that tool calls can be matched to tool responses.


key: _node_content
string_value:
{"id_": "03eec2f3-7071-4178-abc2-dcb07ac9a146", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4ff4d19-54e2-4a5b-9a73-8850dee1aceb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "df28160323be9d9888dc9477f49161f5b7389c8c7c93b018c66c8498e134aa6e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "423d51bd-6503-49a3-b91f-cc1277afbf38", "node_type": "1", "metadata": {}, "hash": "2cfdc4220ee7f7bdfbdc10c194c78551a02fcd8cf80bf58670182e96f5b59620", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16199, "end_char_idx": 19311, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

The output format above is specific to the `Hermes-2-Pro` model we're using in this example. Other models may emit different
tool call formats, and you may need to do some manual parsing at this step. For example, `Llama-3.1` models will emit
slightly different JSON, with `parameters` instead of `arguments`. Regardless of the format the model outputs, you 
should add the tool call to the conversation in the format below, with `tool_calls`, `function` and `arguments` keys. 

</Tip>

Next, let's append the model's tool call to the conversation.

```python
tool_call = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
messages.append({"role": "assistant", "tool_calls": [{"type": "function", "function": tool_call}]})
```


Now that we've added the tool call to the conversation, we can call the function and append the result to the
conversation. Since we're just using a dummy function for this example that always returns 22.0, we can just append 
that result directly.

```python
messages.append({"role": "tool", "name": "get_current_temperature", "content": "22.0"})
```

<Tip>

Some model architectures, notably Mistral/Mixtral, also require a `tool_call_id` here, which should be
9 randomly-generated alphanumeric characters, and assigned to the `id` key of the tool call
dictionary. The same key should also be assigned to the `tool_call_id` key of the tool response dictionary below, so 
that tool calls can be matched to tool responses. So, for Mistral/Mixtral models, the code above would be:

```python
tool_call_id = "9Ae3bDc2F"  # Random ID, 9 alphanumeric characters
tool_call = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
messages.append({"role": "assistant", "tool_calls": [{"type": "function", "id": tool_call_id, "function": tool_call}]})
```

and

```python
messages.append({"role": "tool", "tool_call_id": tool_call_id, "name": "get_current_temperature", "content": "22.0"})
```

</Tip>

Finally, let's let the assistant read the function outputs and continue chatting with the user:

```python
inputs = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_dict=True, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0][len(inputs["input_ids"][0]):]))
```

And we get:

```text
The current temperature in Paris, France is 22.0 ° Celsius.<|im_end|>
```

Although this was a simple demo with dummy tools and a single call, the same technique works with 
multiple real tools and longer conversations. This can be a powerful way to extend the capabilities of conversational
agents with real-time information, computational tools like calculators, or access to large databases.

### Understanding tool schemas

Each function you pass to the `tools` argument of `apply_chat_template` is converted into a 
[JSON schema](https://json-schema.org/learn/getting-started-step-by-step). These schemas
are then passed to the model chat template.


key: _node_content
string_value:
{"id_": "423d51bd-6503-49a3-b91f-cc1277afbf38", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "03eec2f3-7071-4178-abc2-dcb07ac9a146", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "10f76664295da808d7b70a21532ec6a1888edb63a2f918590d77c70a4644679b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a0ffe512-ba07-45d1-afe9-558904a581d7", "node_type": "1", "metadata": {}, "hash": "9632022fb240da2c98495cc2bbe636698fdd5b7832d91472295a21c7abb25885", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17706, "end_char_idx": 20084, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
So, for Mistral/Mixtral models, the code above would be:

```python
tool_call_id = "9Ae3bDc2F"  # Random ID, 9 alphanumeric characters
tool_call = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
messages.append({"role": "assistant", "tool_calls": [{"type": "function", "id": tool_call_id, "function": tool_call}]})
```

and

```python
messages.append({"role": "tool", "tool_call_id": tool_call_id, "name": "get_current_temperature", "content": "22.0"})
```

</Tip>

Finally, let's let the assistant read the function outputs and continue chatting with the user:

```python
inputs = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_dict=True, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0][len(inputs["input_ids"][0]):]))
```

And we get:

```text
The current temperature in Paris, France is 22.0 ° Celsius.<|im_end|>
```

Although this was a simple demo with dummy tools and a single call, the same technique works with 
multiple real tools and longer conversations. This can be a powerful way to extend the capabilities of conversational
agents with real-time information, computational tools like calculators, or access to large databases.

### Understanding tool schemas

Each function you pass to the `tools` argument of `apply_chat_template` is converted into a 
[JSON schema](https://json-schema.org/learn/getting-started-step-by-step). These schemas
are then passed to the model chat template. In other words, tool-use models do not see your functions directly, and they
never see the actual code inside them. What they care about is the function **definitions** and the **arguments** they
need to pass to them - they care about what the tools do and how to use them, not how they work! It is up to you
to read their outputs, detect if they have requested to use a tool, pass their arguments to the tool function, and
return the response in the chat.

Generating JSON schemas to pass to the template should be automatic and invisible as long as your functions
follow the specification above, but if you encounter problems, or you simply want more control over the conversion, 
you can handle the conversion manually. Here is an example of a manual schema conversion.


key: _node_content
string_value:
{"id_": "a0ffe512-ba07-45d1-afe9-558904a581d7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "423d51bd-6503-49a3-b91f-cc1277afbf38", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "c722c34058b28bf5791330d0909b0ff0dc724527174ee2bc28182a0b6ba8dc0c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f3dd02ac-7337-48a4-bb29-0215f66c28ca", "node_type": "1", "metadata": {}, "hash": "04b122433c2f617b76cf3cc481e1e23fefc0e56c46cf8ad4fddc04dfb3e46efd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18878, "end_char_idx": 21752, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This can be a powerful way to extend the capabilities of conversational
agents with real-time information, computational tools like calculators, or access to large databases.

### Understanding tool schemas

Each function you pass to the `tools` argument of `apply_chat_template` is converted into a 
[JSON schema](https://json-schema.org/learn/getting-started-step-by-step). These schemas
are then passed to the model chat template. In other words, tool-use models do not see your functions directly, and they
never see the actual code inside them. What they care about is the function **definitions** and the **arguments** they
need to pass to them - they care about what the tools do and how to use them, not how they work! It is up to you
to read their outputs, detect if they have requested to use a tool, pass their arguments to the tool function, and
return the response in the chat.

Generating JSON schemas to pass to the template should be automatic and invisible as long as your functions
follow the specification above, but if you encounter problems, or you simply want more control over the conversion, 
you can handle the conversion manually. Here is an example of a manual schema conversion.

```python
from transformers.utils import get_json_schema

def multiply(a: float, b: float):
    """
    A function that multiplies two numbers
    
    Args:
        a: The first number to multiply
        b: The second number to multiply
    """
    return a * b

schema = get_json_schema(multiply)
print(schema)
```

This will yield:

```json
{
  "type": "function", 
  "function": {
    "name": "multiply", 
    "description": "A function that multiplies two numbers", 
    "parameters": {
      "type": "object", 
      "properties": {
        "a": {
          "type": "number", 
          "description": "The first number to multiply"
        }, 
        "b": {
          "type": "number",
          "description": "The second number to multiply"
        }
      }, 
      "required": ["a", "b"]
    }
  }
}
```

If you wish, you can edit these schemas, or even write them from scratch yourself without using `get_json_schema` at 
all. JSON schemas can be passed directly to the `tools` argument of 
`apply_chat_template` - this gives you a lot of power to define precise schemas for more complex functions. Be careful,
though - the more complex your schemas, the more likely the model is to get confused when dealing with them! We 
recommend simple function signatures where possible, keeping arguments (and especially complex, nested arguments) 
to a minimum.

Here is an example of defining schemas by hand, and passing them directly to `apply_chat_template`:

```python
# A simple function that takes no arguments
current_time = {
  "type": "function", 
  "function": {
    "name": "current_time",
    "description": "Get the current local time as a string.


key: _node_content
string_value:
{"id_": "f3dd02ac-7337-48a4-bb29-0215f66c28ca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a0ffe512-ba07-45d1-afe9-558904a581d7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "4a273ee825b87a79111fc721284c2a39d82a5e437cd3eeddfbe474a7adf86434", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "602567ac-de5b-42ca-82ad-8452c96904fd", "node_type": "1", "metadata": {}, "hash": "4cca5d89125ec3c144f66865f56ebd168b4054814bb4ea2ce9f279916e52aedc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20086, "end_char_idx": 23249, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers.utils import get_json_schema

def multiply(a: float, b: float):
    """
    A function that multiplies two numbers
    
    Args:
        a: The first number to multiply
        b: The second number to multiply
    """
    return a * b

schema = get_json_schema(multiply)
print(schema)
```

This will yield:

```json
{
  "type": "function", 
  "function": {
    "name": "multiply", 
    "description": "A function that multiplies two numbers", 
    "parameters": {
      "type": "object", 
      "properties": {
        "a": {
          "type": "number", 
          "description": "The first number to multiply"
        }, 
        "b": {
          "type": "number",
          "description": "The second number to multiply"
        }
      }, 
      "required": ["a", "b"]
    }
  }
}
```

If you wish, you can edit these schemas, or even write them from scratch yourself without using `get_json_schema` at 
all. JSON schemas can be passed directly to the `tools` argument of 
`apply_chat_template` - this gives you a lot of power to define precise schemas for more complex functions. Be careful,
though - the more complex your schemas, the more likely the model is to get confused when dealing with them! We 
recommend simple function signatures where possible, keeping arguments (and especially complex, nested arguments) 
to a minimum.

Here is an example of defining schemas by hand, and passing them directly to `apply_chat_template`:

```python
# A simple function that takes no arguments
current_time = {
  "type": "function", 
  "function": {
    "name": "current_time",
    "description": "Get the current local time as a string.",
    "parameters": {
      'type': 'object',
      'properties': {}
    }
  }
}

# A more complete function that takes two numerical arguments
multiply = {
  'type': 'function',
  'function': {
    'name': 'multiply',
    'description': 'A function that multiplies two numbers', 
    'parameters': {
      'type': 'object', 
      'properties': {
        'a': {
          'type': 'number',
          'description': 'The first number to multiply'
        }, 
        'b': {
          'type': 'number', 'description': 'The second number to multiply'
        }
      }, 
      'required': ['a', 'b']
    }
  }
}

model_input = tokenizer.apply_chat_template(
    messages,
    tools = [current_time, multiply]
)
```

## Advanced: Retrieval-augmented generation

"Retrieval-augmented generation" or "RAG" LLMs can search a corpus of documents for information before responding
to a query. This allows models to vastly expand their knowledge base beyond their limited context size. Our 
recommendation for RAG models is that their template
should accept a `documents` argument. This should be a list of documents, where each "document"
is a single dict with `title` and `contents` keys, both of which are strings. Because this format is much simpler
than the JSON schemas used for tools, no helper functions are necessary.

Here's an example of a RAG template in action:

```python
document1 = {
    "title": "The Moon: Our Age-Old Foe",
    "contents": "Man has always dreamed of destroying the moon.


key: _node_content
string_value:
{"id_": "602567ac-de5b-42ca-82ad-8452c96904fd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f3dd02ac-7337-48a4-bb29-0215f66c28ca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "a020625ea79032796e66f4118fbc070652e8dcd98f6a47b48fa37862c674f9ed", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9bd2054-200d-4a61-9687-77aa2063c3f1", "node_type": "1", "metadata": {}, "hash": "b9957649c918f82549f1288887196479431035b7b872f14ae4124ec8b731dc7c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21752, "end_char_idx": 24807, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
    "parameters": {
      'type': 'object',
      'properties': {}
    }
  }
}

# A more complete function that takes two numerical arguments
multiply = {
  'type': 'function',
  'function': {
    'name': 'multiply',
    'description': 'A function that multiplies two numbers', 
    'parameters': {
      'type': 'object', 
      'properties': {
        'a': {
          'type': 'number',
          'description': 'The first number to multiply'
        }, 
        'b': {
          'type': 'number', 'description': 'The second number to multiply'
        }
      }, 
      'required': ['a', 'b']
    }
  }
}

model_input = tokenizer.apply_chat_template(
    messages,
    tools = [current_time, multiply]
)
```

## Advanced: Retrieval-augmented generation

"Retrieval-augmented generation" or "RAG" LLMs can search a corpus of documents for information before responding
to a query. This allows models to vastly expand their knowledge base beyond their limited context size. Our 
recommendation for RAG models is that their template
should accept a `documents` argument. This should be a list of documents, where each "document"
is a single dict with `title` and `contents` keys, both of which are strings. Because this format is much simpler
than the JSON schemas used for tools, no helper functions are necessary.

Here's an example of a RAG template in action:

```python
document1 = {
    "title": "The Moon: Our Age-Old Foe",
    "contents": "Man has always dreamed of destroying the moon. In this essay, I shall..."
}

document2 = {
    "title": "The Sun: Our Age-Old Friend",
    "contents": "Although often underappreciated, the sun provides several notable benefits..."
}

model_input = tokenizer.apply_chat_template(
    messages,
    documents=[document1, document2]
)
```

## Advanced: How do chat templates work?

The chat template for a model is stored on the `tokenizer.chat_template` attribute. If no chat template is set, the
default template for that model class is used instead. Let's take a look at the template for `BlenderBot`:

```python

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")

>>> tokenizer.chat_template
"{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ '  ' }}{% endif %}{% endfor %}{{ eos_token }}"
```

That's kind of intimidating. Let's clean it up a little to make it more readable. In the process, though, we also make
sure that the newlines and indentation we add don't end up being included in the template output - see the tip on
[trimming whitespace](#trimming-whitespace) below!

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- ' ' }}
    {%- endif %}
    {{- message['content'] }}
    {%- if not loop.last %}
        {{- '  ' }}
    {%- endif %}
{%- endfor %}
{{- eos_token }}
```

If you've never seen one of these before, this is a [Jinja template](https://jinja.palletsprojects.com/en/3.1.x/templates/).


key: _node_content
string_value:
{"id_": "a9bd2054-200d-4a61-9687-77aa2063c3f1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "602567ac-de5b-42ca-82ad-8452c96904fd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "dd8dc74bf2f4dfa56cc3fd20d7c202bb811fe7e5d5becc0061faf7894f88014e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fcbc8aec-1561-4256-af0f-86a0f75985d7", "node_type": "1", "metadata": {}, "hash": "758f1b8e3dcb8fc9afe129f6507242ba3ccb97b08e8454439838e7227e8aae66", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23582, "end_char_idx": 26127, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The chat template for a model is stored on the `tokenizer.chat_template` attribute. If no chat template is set, the
default template for that model class is used instead. Let's take a look at the template for `BlenderBot`:

```python

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")

>>> tokenizer.chat_template
"{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ '  ' }}{% endif %}{% endfor %}{{ eos_token }}"
```

That's kind of intimidating. Let's clean it up a little to make it more readable. In the process, though, we also make
sure that the newlines and indentation we add don't end up being included in the template output - see the tip on
[trimming whitespace](#trimming-whitespace) below!

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- ' ' }}
    {%- endif %}
    {{- message['content'] }}
    {%- if not loop.last %}
        {{- '  ' }}
    {%- endif %}
{%- endfor %}
{{- eos_token }}
```

If you've never seen one of these before, this is a [Jinja template](https://jinja.palletsprojects.com/en/3.1.x/templates/).
Jinja is a templating language that allows you to write simple code that generates text. In many ways, the code and
syntax resembles Python. In pure Python, this template would look something like this:

```python
for idx, message in enumerate(messages):
    if message['role'] == 'user':
        print(' ')
    print(message['content'])
    if not idx == len(messages) - 1:  # Check for the last message in the conversation
        print('  ')
print(eos_token)
```

Effectively, the template does three things:
1. For each message, if the message is a user message, add a blank space before it, otherwise print nothing.
2. Add the message content
3. If the message is not the last message, add two spaces after it. After the final message, print the EOS token.

This is a pretty simple template - it doesn't add any control tokens, and it doesn't support "system" messages, which 
are a common way to give the model directives about how it should behave in the subsequent conversation.
But Jinja gives you a lot of flexibility to do those things! Let's see a Jinja template that can format inputs
similarly to the way LLaMA formats them (note that the real LLaMA template includes handling for default system
messages and slightly different system message handling in general - don't use this one in your actual code!)


key: _node_content
string_value:
{"id_": "fcbc8aec-1561-4256-af0f-86a0f75985d7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9bd2054-200d-4a61-9687-77aa2063c3f1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "881c3f6fc15d2eb0db78534f089748a7d5fb2dbe31d1d95a8ddc2e2e9aea4f1c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c03e4a3-8e7e-4f9f-986a-5bd8411618a4", "node_type": "1", "metadata": {}, "hash": "270533b1c4006fc61fde43555ea9cbabb15a108a44d25cae80851f304a81defd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24808, "end_char_idx": 27793, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Jinja is a templating language that allows you to write simple code that generates text. In many ways, the code and
syntax resembles Python. In pure Python, this template would look something like this:

```python
for idx, message in enumerate(messages):
    if message['role'] == 'user':
        print(' ')
    print(message['content'])
    if not idx == len(messages) - 1:  # Check for the last message in the conversation
        print('  ')
print(eos_token)
```

Effectively, the template does three things:
1. For each message, if the message is a user message, add a blank space before it, otherwise print nothing.
2. Add the message content
3. If the message is not the last message, add two spaces after it. After the final message, print the EOS token.

This is a pretty simple template - it doesn't add any control tokens, and it doesn't support "system" messages, which 
are a common way to give the model directives about how it should behave in the subsequent conversation.
But Jinja gives you a lot of flexibility to do those things! Let's see a Jinja template that can format inputs
similarly to the way LLaMA formats them (note that the real LLaMA template includes handling for default system
messages and slightly different system message handling in general - don't use this one in your actual code!)

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- bos_token + '[INST] ' + message['content'] + ' [/INST]' }}
    {%- elif message['role'] == 'system' %}
        {{- '<<SYS>>\\n' + message['content'] + '\\n<</SYS>>\\n\\n' }}
    {%- elif message['role'] == 'assistant' %}
        {{- ' '  + message['content'] + ' ' + eos_token }}
    {%- endif %}
{%- endfor %}
```

Hopefully if you stare at this for a little bit you can see what this template is doing - it adds specific tokens based
on the "role" of each message, which represents who sent it. User, assistant and system messages are clearly
distinguishable to the model because of the tokens they're wrapped in.

## Advanced: Adding and editing chat templates

### How do I create a chat template?

Simple, just write a jinja template and set `tokenizer.chat_template`. You may find it easier to start with an 
existing template from another model and simply edit it for your needs! For example, we could take the LLaMA template
above and add "[ASST]" and "[/ASST]" to assistant messages:

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- bos_token + '[INST] ' + message['content'].strip() + ' [/INST]' }}
    {%- elif message['role'] == 'system' %}
        {{- '<<SYS>>\\n' + message['content'].strip() + '\\n<</SYS>>\\n\\n' }}
    {%- elif message['role'] == 'assistant' %}
        {{- '[ASST] '  + message['content'] + ' [/ASST]' + eos_token }}
    {%- endif %}
{%- endfor %}
```

Now, simply set the `tokenizer.chat_template` attribute. Next time you use [`~PreTrainedTokenizer.apply_chat_template`], it will
use your new template!


key: _node_content
string_value:
{"id_": "7c03e4a3-8e7e-4f9f-986a-5bd8411618a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fcbc8aec-1561-4256-af0f-86a0f75985d7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "7c3df6890dc2251d7a4abc5ba8beb32308787bebf1c894fc719fb6664665cabb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "43913576-ec5a-4d3b-b495-a2b7793e2654", "node_type": "1", "metadata": {}, "hash": "043e4963ac31fab00c5ce61db3f70b730089dbe3815a0f4332c0450a859f16e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26713, "end_char_idx": 30080, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
User, assistant and system messages are clearly
distinguishable to the model because of the tokens they're wrapped in.

## Advanced: Adding and editing chat templates

### How do I create a chat template?

Simple, just write a jinja template and set `tokenizer.chat_template`. You may find it easier to start with an 
existing template from another model and simply edit it for your needs! For example, we could take the LLaMA template
above and add "[ASST]" and "[/ASST]" to assistant messages:

```
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- bos_token + '[INST] ' + message['content'].strip() + ' [/INST]' }}
    {%- elif message['role'] == 'system' %}
        {{- '<<SYS>>\\n' + message['content'].strip() + '\\n<</SYS>>\\n\\n' }}
    {%- elif message['role'] == 'assistant' %}
        {{- '[ASST] '  + message['content'] + ' [/ASST]' + eos_token }}
    {%- endif %}
{%- endfor %}
```

Now, simply set the `tokenizer.chat_template` attribute. Next time you use [`~PreTrainedTokenizer.apply_chat_template`], it will
use your new template! This attribute will be saved in the `tokenizer_config.json` file, so you can use
[`~utils.PushToHubMixin.push_to_hub`] to upload your new template to the Hub and make sure everyone's using the right
template for your model!

```python
template = tokenizer.chat_template
template = template.replace("SYS", "SYSTEM")  # Change the system token
tokenizer.chat_template = template  # Set the new template
tokenizer.push_to_hub("model_name")  # Upload your new template to the Hub!
```

The method [`~PreTrainedTokenizer.apply_chat_template`] which uses your chat template is called by the [`TextGenerationPipeline`] class, so 
once you set the correct chat template, your model will automatically become compatible with [`TextGenerationPipeline`].

<Tip>
If you're fine-tuning a model for chat, in addition to setting a chat template, you should probably add any new chat
control tokens as special tokens in the tokenizer. Special tokens are never split, 
ensuring that your control tokens are always handled as single tokens rather than being tokenized in pieces. You 
should also set the tokenizer's `eos_token` attribute to the token that marks the end of assistant generations in your
template. This will ensure that text generation tools can correctly figure out when to stop generating text.
</Tip>


### Why do some models have multiple templates?

Some models use different templates for different use cases. For example, they might use one template for normal chat
and another for tool-use, or retrieval-augmented generation. In these cases, `tokenizer.chat_template` is a dictionary.
This can cause some confusion, and where possible, we recommend using a single template for all use-cases. You can use
Jinja statements like `if tools is defined` and `{% macro %}` definitions to easily wrap multiple code paths in a
single template.

When a tokenizer has multiple templates, `tokenizer.chat_template` will be a `dict`, where each key is the name
of a template. The `apply_chat_template` method has special handling for certain template names: Specifically, it will
look for a template named `default` in most cases, and will raise an error if it can't find one. However, if a template
named `tool_use` exists when the user has passed a `tools` argument, it will use that instead.


key: _node_content
string_value:
{"id_": "43913576-ec5a-4d3b-b495-a2b7793e2654", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c03e4a3-8e7e-4f9f-986a-5bd8411618a4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "d44313e4f24ce6b97c0fa5fd81a361df95d58ef631bec21f32e36ffcbdc7eef2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad292bb9-2cd5-434e-9ee3-e15e2bcbae76", "node_type": "1", "metadata": {}, "hash": "59985f350bc611905b37d72e60b0e15cdc0e7e24a242f61ec8c5bba498e44a4a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28271, "end_char_idx": 31727, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

The method [`~PreTrainedTokenizer.apply_chat_template`] which uses your chat template is called by the [`TextGenerationPipeline`] class, so 
once you set the correct chat template, your model will automatically become compatible with [`TextGenerationPipeline`].

<Tip>
If you're fine-tuning a model for chat, in addition to setting a chat template, you should probably add any new chat
control tokens as special tokens in the tokenizer. Special tokens are never split, 
ensuring that your control tokens are always handled as single tokens rather than being tokenized in pieces. You 
should also set the tokenizer's `eos_token` attribute to the token that marks the end of assistant generations in your
template. This will ensure that text generation tools can correctly figure out when to stop generating text.
</Tip>


### Why do some models have multiple templates?

Some models use different templates for different use cases. For example, they might use one template for normal chat
and another for tool-use, or retrieval-augmented generation. In these cases, `tokenizer.chat_template` is a dictionary.
This can cause some confusion, and where possible, we recommend using a single template for all use-cases. You can use
Jinja statements like `if tools is defined` and `{% macro %}` definitions to easily wrap multiple code paths in a
single template.

When a tokenizer has multiple templates, `tokenizer.chat_template` will be a `dict`, where each key is the name
of a template. The `apply_chat_template` method has special handling for certain template names: Specifically, it will
look for a template named `default` in most cases, and will raise an error if it can't find one. However, if a template
named `tool_use` exists when the user has passed a `tools` argument, it will use that instead. To access templates
with other names, pass the name of the template you want to the `chat_template` argument of
`apply_chat_template()`.

We find that this can be a bit confusing for users, though - so if you're writing a template yourself, we recommend
trying to put it all in a single template where possible!

### What template should I use?

When setting the template for a model that's already been trained for chat, you should ensure that the template
exactly matches the message formatting that the model saw during training, or else you will probably experience
performance degradation. This is true even if you're training the model further - you will probably get the best 
performance if you keep the chat tokens constant. This is very analogous to tokenization - you generally get the
best performance for inference or fine-tuning when you precisely match the tokenization used during training.

If you're training a model from scratch, or fine-tuning a base language model for chat, on the other hand,
you have a lot of freedom to choose an appropriate template! LLMs are smart enough to learn to handle lots of different
input formats. One popular choice is the `ChatML` format, and this is a good, flexible choice for many use-cases. 
It looks like this:

```
{%- for message in messages %}
    {{- '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n' }}
{%- endfor %}
```

If you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes
handy support for [generation prompts](#what-are-generation-prompts), but note that it doesn't add BOS or EOS tokens!


key: _node_content
string_value:
{"id_": "ad292bb9-2cd5-434e-9ee3-e15e2bcbae76", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "43913576-ec5a-4d3b-b495-a2b7793e2654", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "d28e2c13028a1f7386b90b570e95a014d43d5473f6a78eafaa4ee361f43930bd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b6b14e35-a6a6-4721-bf82-07caf2723993", "node_type": "1", "metadata": {}, "hash": "c82d849db6365dde44636fd53d64844802c7b76a5625b77d7a268500f781bf7b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30081, "end_char_idx": 33223, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To access templates
with other names, pass the name of the template you want to the `chat_template` argument of
`apply_chat_template()`.

We find that this can be a bit confusing for users, though - so if you're writing a template yourself, we recommend
trying to put it all in a single template where possible!

### What template should I use?

When setting the template for a model that's already been trained for chat, you should ensure that the template
exactly matches the message formatting that the model saw during training, or else you will probably experience
performance degradation. This is true even if you're training the model further - you will probably get the best 
performance if you keep the chat tokens constant. This is very analogous to tokenization - you generally get the
best performance for inference or fine-tuning when you precisely match the tokenization used during training.

If you're training a model from scratch, or fine-tuning a base language model for chat, on the other hand,
you have a lot of freedom to choose an appropriate template! LLMs are smart enough to learn to handle lots of different
input formats. One popular choice is the `ChatML` format, and this is a good, flexible choice for many use-cases. 
It looks like this:

```
{%- for message in messages %}
    {{- '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n' }}
{%- endfor %}
```

If you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes
handy support for [generation prompts](#what-are-generation-prompts), but note that it doesn't add BOS or EOS tokens!
If your model expects those, they won't be added automatically by `apply_chat_template` - in other words, the
text will be tokenized with `add_special_tokens=False`. This is to avoid potential conflicts between the template and
the `add_special_tokens` logic. If your model expects special tokens, make sure to add them to the template!

```python
tokenizer.chat_template = "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
```

This template wraps each message in `<|im_start|>` and `<|im_end|>` tokens, and simply writes the role as a string, which
allows for flexibility in the roles you train with. The output looks like this:

```text
<|im_start|>system
You are a helpful chatbot that will do its best not to say anything so stupid that people tweet about it.<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
I'm doing great!<|im_end|>
```

The "user", "system" and "assistant" roles are the standard for chat, and we recommend using them when it makes sense,
particularly if you want your model to operate well with [`TextGenerationPipeline`]. However, you are not limited
to these roles - templating is extremely flexible, and any string can be a role.

### I want to add some chat templates! How should I get started?


key: _node_content
string_value:
{"id_": "b6b14e35-a6a6-4721-bf82-07caf2723993", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad292bb9-2cd5-434e-9ee3-e15e2bcbae76", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "9b13a953ba3a1514d87e770a3f874c689e745787458ed7908132a634334fbd52", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dc64fb79-4700-4724-b4a1-de09de85098a", "node_type": "1", "metadata": {}, "hash": "a7e8332f7084876925023b8481bf37b7beac0ba579864ebc34ab688aea4fc41a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31728, "end_char_idx": 34905, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If your model expects those, they won't be added automatically by `apply_chat_template` - in other words, the
text will be tokenized with `add_special_tokens=False`. This is to avoid potential conflicts between the template and
the `add_special_tokens` logic. If your model expects special tokens, make sure to add them to the template!

```python
tokenizer.chat_template = "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
```

This template wraps each message in `<|im_start|>` and `<|im_end|>` tokens, and simply writes the role as a string, which
allows for flexibility in the roles you train with. The output looks like this:

```text
<|im_start|>system
You are a helpful chatbot that will do its best not to say anything so stupid that people tweet about it.<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
I'm doing great!<|im_end|>
```

The "user", "system" and "assistant" roles are the standard for chat, and we recommend using them when it makes sense,
particularly if you want your model to operate well with [`TextGenerationPipeline`]. However, you are not limited
to these roles - templating is extremely flexible, and any string can be a role.

### I want to add some chat templates! How should I get started?

If you have any chat models, you should set their `tokenizer.chat_template` attribute and test it using
[`~PreTrainedTokenizer.apply_chat_template`], then push the updated tokenizer to the Hub. This applies even if you're
not the model owner - if you're using a model with an empty chat template, or one that's still using the default class
template, please open a [pull request](https://huggingface.co/docs/hub/repositories-pull-requests-discussions) to the model repository so that this attribute can be set properly!

Once the attribute is set, that's it, you're done! `tokenizer.apply_chat_template` will now work correctly for that
model, which means it is also automatically supported in places like `TextGenerationPipeline`!

By ensuring that models have this attribute, we can make sure that the whole community gets to use the full power of
open-source models. Formatting mismatches have been haunting the field and silently harming performance for too long - 
it's time to put an end to them!

## Advanced: Template writing tips

<Tip>

The easiest way to get started with writing Jinja templates is to take a look at some existing ones. You can use
`print(tokenizer.chat_template)` for any chat model to see what template it's using. In general, models that support tool use have 
much more complex templates than other models - so when you're just getting started, they're probably a bad example
to learn from! You can also take a look at the 
[Jinja documentation](https://jinja.palletsprojects.com/en/3.1.x/templates/#synopsis) for details
of general Jinja formatting and syntax.

</Tip>

Jinja templates in `transformers` are identical to Jinja templates elsewhere.


key: _node_content
string_value:
{"id_": "dc64fb79-4700-4724-b4a1-de09de85098a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b6b14e35-a6a6-4721-bf82-07caf2723993", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "aba2e1154d3044c78057e2633728fc827ff1802caaea454f78283c06c19e4df5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "04b9f14d-7a45-45fa-b3e7-10d6d12dc57d", "node_type": "1", "metadata": {}, "hash": "7667bf17a9c09adacc8ab19588a41e0f6144d94e452b065a1a7a1a70c6a008b1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33159, "end_char_idx": 36630, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### I want to add some chat templates! How should I get started?

If you have any chat models, you should set their `tokenizer.chat_template` attribute and test it using
[`~PreTrainedTokenizer.apply_chat_template`], then push the updated tokenizer to the Hub. This applies even if you're
not the model owner - if you're using a model with an empty chat template, or one that's still using the default class
template, please open a [pull request](https://huggingface.co/docs/hub/repositories-pull-requests-discussions) to the model repository so that this attribute can be set properly!

Once the attribute is set, that's it, you're done! `tokenizer.apply_chat_template` will now work correctly for that
model, which means it is also automatically supported in places like `TextGenerationPipeline`!

By ensuring that models have this attribute, we can make sure that the whole community gets to use the full power of
open-source models. Formatting mismatches have been haunting the field and silently harming performance for too long - 
it's time to put an end to them!

## Advanced: Template writing tips

<Tip>

The easiest way to get started with writing Jinja templates is to take a look at some existing ones. You can use
`print(tokenizer.chat_template)` for any chat model to see what template it's using. In general, models that support tool use have 
much more complex templates than other models - so when you're just getting started, they're probably a bad example
to learn from! You can also take a look at the 
[Jinja documentation](https://jinja.palletsprojects.com/en/3.1.x/templates/#synopsis) for details
of general Jinja formatting and syntax.

</Tip>

Jinja templates in `transformers` are identical to Jinja templates elsewhere. The main thing to know is that 
the conversation history will be accessible inside your template as a variable called `messages`.  
You will be able to access `messages` in your template just like you can in Python, which means you can loop over 
it with `{% for message in messages %}` or access individual messages with `{{ messages[0] }}`, for example.

You can also use the following tips to write clean, efficient Jinja templates:

### Trimming whitespace

By default, Jinja will print any whitespace that comes before or after a block. This can be a problem for chat
templates, which generally want to be very precise with whitespace! To avoid this, we strongly recommend writing
your templates like this:

```
{%- for message in messages %}
    {{- message['role'] + message['content'] }}
{%- endfor %}
```

rather than like this:

```
{% for message in messages %}
    {{ message['role'] + message['content'] }}
{% endfor %}
```

Adding `-` will strip any whitespace that comes before the block. The second example looks innocent, but the newline
and indentation may end up being included in the output, which is probably not what you want!

### Special variables

Inside your template, you will have access several special variables. The most important of these is `messages`, 
which contains the chat history as a list of message dicts. However, there are several others. Not every
variable will be used in every template. The most common other variables are:

- `tools` contains a list of tools in JSON schema format. Will be `None` or undefined if no tools are passed.
- `documents` contains a list of documents in the format `{"title": "Title", "contents": "Contents"}`, used for retrieval-augmented generation.


key: _node_content
string_value:
{"id_": "04b9f14d-7a45-45fa-b3e7-10d6d12dc57d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dc64fb79-4700-4724-b4a1-de09de85098a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "00045fb4b6bb7ed8165ec8c651536a03703186bf03b35711e3fc2bc21b468b51", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "048abccb-0e06-4f93-b102-8ae8e1907cc0", "node_type": "1", "metadata": {}, "hash": "f035e37d1877f06bfe403b4a9d01cebdd8b5444ae127f43e3484b33cf709b8a3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34906, "end_char_idx": 38423, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The main thing to know is that 
the conversation history will be accessible inside your template as a variable called `messages`.  
You will be able to access `messages` in your template just like you can in Python, which means you can loop over 
it with `{% for message in messages %}` or access individual messages with `{{ messages[0] }}`, for example.

You can also use the following tips to write clean, efficient Jinja templates:

### Trimming whitespace

By default, Jinja will print any whitespace that comes before or after a block. This can be a problem for chat
templates, which generally want to be very precise with whitespace! To avoid this, we strongly recommend writing
your templates like this:

```
{%- for message in messages %}
    {{- message['role'] + message['content'] }}
{%- endfor %}
```

rather than like this:

```
{% for message in messages %}
    {{ message['role'] + message['content'] }}
{% endfor %}
```

Adding `-` will strip any whitespace that comes before the block. The second example looks innocent, but the newline
and indentation may end up being included in the output, which is probably not what you want!

### Special variables

Inside your template, you will have access several special variables. The most important of these is `messages`, 
which contains the chat history as a list of message dicts. However, there are several others. Not every
variable will be used in every template. The most common other variables are:

- `tools` contains a list of tools in JSON schema format. Will be `None` or undefined if no tools are passed.
- `documents` contains a list of documents in the format `{"title": "Title", "contents": "Contents"}`, used for retrieval-augmented generation. Will be `None` or undefined if no documents are passed.
- `add_generation_prompt` is a bool that is `True` if the user has requested a generation prompt, and `False` otherwise. If this is set, your template should add the header for an assistant message to the end of the conversation. If your model doesn't have a specific header for assistant messages, you can ignore this flag.
- **Special tokens** like `bos_token` and `eos_token`. These are extracted from `tokenizer.special_tokens_map`. The exact tokens available inside each template will differ depending on the parent tokenizer.

<Tip>

You can actually pass any `kwarg` to `apply_chat_template`, and it will be accessible inside the template as a variable. In general,
we recommend trying to stick to the core variables above, as it will make your model harder to use if users have
to write custom code to pass model-specific `kwargs`. However, we're aware that this field moves quickly, so if you
have a new use-case that doesn't fit in the core API, feel free to use a new `kwarg` for it! If a new `kwarg`
becomes common we may promote it into the core API and create a standard, documented format for it.

</Tip>

### Callable functions

There is also a short list of callable functions available to you inside your templates. These are:

- `raise_exception(msg)`: Raises a `TemplateException`. This is useful for debugging, and for telling users when they're
doing something that your template doesn't support.
- `strftime_now(format_str)`: Equivalent to `datetime.now().strftime(format_str)` in Python. This is used for getting
the current date/time in a specific format, which is sometimes included in system messages.

### Compatibility with non-Python Jinja

There are multiple implementations of Jinja in various languages.


key: _node_content
string_value:
{"id_": "048abccb-0e06-4f93-b102-8ae8e1907cc0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "04b9f14d-7a45-45fa-b3e7-10d6d12dc57d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "9140df0a93c75a901cdacb64ed70bfc6cbd650ab07b22846ccfdf560c38a3e9a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "45f5974e-f938-4b4e-ad12-0b36ab9135b4", "node_type": "1", "metadata": {}, "hash": "27a068cf57962ecc20e774f50b68746e7cf4e5a1c6439fcb63a7931b622ddd1a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36631, "end_char_idx": 40066, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Will be `None` or undefined if no documents are passed.
- `add_generation_prompt` is a bool that is `True` if the user has requested a generation prompt, and `False` otherwise. If this is set, your template should add the header for an assistant message to the end of the conversation. If your model doesn't have a specific header for assistant messages, you can ignore this flag.
- **Special tokens** like `bos_token` and `eos_token`. These are extracted from `tokenizer.special_tokens_map`. The exact tokens available inside each template will differ depending on the parent tokenizer.

<Tip>

You can actually pass any `kwarg` to `apply_chat_template`, and it will be accessible inside the template as a variable. In general,
we recommend trying to stick to the core variables above, as it will make your model harder to use if users have
to write custom code to pass model-specific `kwargs`. However, we're aware that this field moves quickly, so if you
have a new use-case that doesn't fit in the core API, feel free to use a new `kwarg` for it! If a new `kwarg`
becomes common we may promote it into the core API and create a standard, documented format for it.

</Tip>

### Callable functions

There is also a short list of callable functions available to you inside your templates. These are:

- `raise_exception(msg)`: Raises a `TemplateException`. This is useful for debugging, and for telling users when they're
doing something that your template doesn't support.
- `strftime_now(format_str)`: Equivalent to `datetime.now().strftime(format_str)` in Python. This is used for getting
the current date/time in a specific format, which is sometimes included in system messages.

### Compatibility with non-Python Jinja

There are multiple implementations of Jinja in various languages. They generally have the same syntax,
but a key difference is that when you're writing a template in Python you can use Python methods, such as
`.lower()` on strings or `.items()` on dicts. This will break if someone tries to use your template on a non-Python
implementation of Jinja. Non-Python implementations are particularly common in deployment environments, where JS
and Rust are very popular. 

Don't panic, though! There are a few easy changes you can make to your templates to ensure they're compatible across
all implementations of Jinja:

- Replace Python methods with Jinja filters. These usually have the same name, for example `string.lower()` becomes
  `string|lower`, and `dict.items()` becomes `dict|items`. One notable change is that `string.strip()` becomes `string|trim`.
  See the [list of built-in filters](https://jinja.palletsprojects.com/en/3.1.x/templates/#builtin-filters)
  in the Jinja documentation for more.
- Replace `True`, `False` and `None`, which are Python-specific, with `true`, `false` and `none`.
- Directly rendering a dict or list may give different results in other implementations (for example, string entries
  might change from single-quoted to double-quoted). Adding the `tojson` filter can help to ensure consistency here.

### Writing and debugging larger templates

When this feature was introduced, most templates were quite small, the Jinja equivalent of a "one-liner" script. 
However, with new models and features like tool-use and RAG, some templates can be 100 lines long or more. When
writing templates like these, it's a good idea to write them in a separate file, using a text editor.


key: _node_content
string_value:
{"id_": "45f5974e-f938-4b4e-ad12-0b36ab9135b4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74fed18e-7830-4a93-a841-924de72c1075", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "18a5d934f38039d7a313e9dde7f6001e9cdfbfb9332c16f472c4d3bf3881667d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "048abccb-0e06-4f93-b102-8ae8e1907cc0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Chat Templates", "tokens": 9701, "retrieve_doc": false, "source": "transformers"}, "hash": "2ce77f812c9f24c8c452449f66bed644dc5333ecc3fe6f2c8933e44dad04e2be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38317, "end_char_idx": 40570, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Compatibility with non-Python Jinja

There are multiple implementations of Jinja in various languages. They generally have the same syntax,
but a key difference is that when you're writing a template in Python you can use Python methods, such as
`.lower()` on strings or `.items()` on dicts. This will break if someone tries to use your template on a non-Python
implementation of Jinja. Non-Python implementations are particularly common in deployment environments, where JS
and Rust are very popular. 

Don't panic, though! There are a few easy changes you can make to your templates to ensure they're compatible across
all implementations of Jinja:

- Replace Python methods with Jinja filters. These usually have the same name, for example `string.lower()` becomes
  `string|lower`, and `dict.items()` becomes `dict|items`. One notable change is that `string.strip()` becomes `string|trim`.
  See the [list of built-in filters](https://jinja.palletsprojects.com/en/3.1.x/templates/#builtin-filters)
  in the Jinja documentation for more.
- Replace `True`, `False` and `None`, which are Python-specific, with `true`, `false` and `none`.
- Directly rendering a dict or list may give different results in other implementations (for example, string entries
  might change from single-quoted to double-quoted). Adding the `tojson` filter can help to ensure consistency here.

### Writing and debugging larger templates

When this feature was introduced, most templates were quite small, the Jinja equivalent of a "one-liner" script. 
However, with new models and features like tool-use and RAG, some templates can be 100 lines long or more. When
writing templates like these, it's a good idea to write them in a separate file, using a text editor. You can easily 
extract a chat template to a file:

```python
open("template.jinja", "w").write(tokenizer.chat_template)
```

Or load the edited template back into the tokenizer:

```python
tokenizer.chat_template = open("template.jinja").read()
```

As an added bonus, when you write a long, multi-line template in a separate file, line numbers in that file will
exactly correspond to line numbers in template parsing or execution errors. This will make it much easier to
identify the source of issues.


key: _node_content
string_value:
{"id_": "074a663d-7510-4d6e-bbdb-d22ccf9a7f6c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "73b2207f-dd9b-495a-8965-7ce9e62e2203", "node_type": "1", "metadata": {}, "hash": "b226a18b18a0ff2715c0422af7dd5128eb99b351bdf1dbb5c8c2fce313f98236", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3234, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to add a model to 🤗 Transformers?

The 🤗 Transformers library is often able to offer new models thanks to community contributors. But this can be a challenging project and requires an in-depth knowledge of the 🤗 Transformers library and the model to implement. At Hugging Face, we're trying to empower more of the community to actively add models and we've put together this guide to walk you through the process of adding a PyTorch model (make sure you have [PyTorch installed](https://pytorch.org/get-started/locally/)).

Along the way, you'll:

- get insights into open-source best practices
- understand the design principles behind one of the most popular deep learning libraries
- learn how to efficiently test large models
- learn how to integrate Python utilities like `black`, `ruff`, and `make fix-copies` to ensure clean and readable code

A Hugging Face team member will be available to help you along the way so you'll never be alone. 🤗 ❤️

To get started, open a [New model addition](https://github.com/huggingface/transformers/issues/new?assignees=&labels=New+model&template=new-model-addition.yml) issue for the model you want to see in 🤗 Transformers. If you're not especially picky about contributing a specific model, you can filter by the [New model label](https://github.com/huggingface/transformers/labels/New%20model) to see if there are any unclaimed model requests and work on it.

Once you've opened a new model request, the first step is to get familiar with 🤗 Transformers if you aren't already!

## General overview of 🤗 Transformers

First, you should get a general overview of 🤗 Transformers. 🤗 Transformers is a very opinionated library, so there is a
chance that you don't agree with some of the library's philosophies or design choices. From our experience, however, we
found that the fundamental design choices and philosophies of the library are crucial to efficiently scale 🤗
Transformers while keeping maintenance costs at a reasonable level.

A good first starting point to better understand the library is to read the [documentation of our philosophy](philosophy). As a result of our way of working, there are some choices that we try to apply to all models:

- Composition is generally favored over-abstraction
- Duplicating code is not always bad if it strongly improves the readability or accessibility of a model
- Model files are as self-contained as possible so that when you read the code of a specific model, you ideally only
  have to look into the respective `modeling_....py` file.

In our opinion, the library's code is not just a means to provide a product, *e.g.* the ability to use BERT for
inference, but also as the very product that we want to improve. Hence, when adding a model, the user is not only the
person who will use your model, but also everybody who will read, try to understand, and possibly tweak your code.

With this in mind, let's go a bit deeper into the general library design.

### Overview of models

To successfully add a model, it is important to understand the interaction between your model and its config,
[`PreTrainedModel`], and [`PretrainedConfig`]. For exemplary purposes, we will
call the model to be added to 🤗 Transformers `BrandNewBert`.


key: _node_content
string_value:
{"id_": "73b2207f-dd9b-495a-8965-7ce9e62e2203", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "074a663d-7510-4d6e-bbdb-d22ccf9a7f6c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "9fb421124cef8f0c531b3296d23d73749bc33d19139382c97cde2be697555702", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bc62741c-9e8f-4154-b2af-eb946bdb517c", "node_type": "1", "metadata": {}, "hash": "55e340fc7b813885869bdc0a62d5f54ce62ce1b90e1ffc6c2fece981c6c0e784", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1531, "end_char_idx": 4527, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## General overview of 🤗 Transformers

First, you should get a general overview of 🤗 Transformers. 🤗 Transformers is a very opinionated library, so there is a
chance that you don't agree with some of the library's philosophies or design choices. From our experience, however, we
found that the fundamental design choices and philosophies of the library are crucial to efficiently scale 🤗
Transformers while keeping maintenance costs at a reasonable level.

A good first starting point to better understand the library is to read the [documentation of our philosophy](philosophy). As a result of our way of working, there are some choices that we try to apply to all models:

- Composition is generally favored over-abstraction
- Duplicating code is not always bad if it strongly improves the readability or accessibility of a model
- Model files are as self-contained as possible so that when you read the code of a specific model, you ideally only
  have to look into the respective `modeling_....py` file.

In our opinion, the library's code is not just a means to provide a product, *e.g.* the ability to use BERT for
inference, but also as the very product that we want to improve. Hence, when adding a model, the user is not only the
person who will use your model, but also everybody who will read, try to understand, and possibly tweak your code.

With this in mind, let's go a bit deeper into the general library design.

### Overview of models

To successfully add a model, it is important to understand the interaction between your model and its config,
[`PreTrainedModel`], and [`PretrainedConfig`]. For exemplary purposes, we will
call the model to be added to 🤗 Transformers `BrandNewBert`.

Let's take a look:

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers_overview.png"/>

As you can see, we do make use of inheritance in 🤗 Transformers, but we keep the level of abstraction to an absolute
minimum. There are never more than two levels of abstraction for any model in the library. `BrandNewBertModel`
inherits from `BrandNewBertPreTrainedModel` which in turn inherits from [`PreTrainedModel`] and
that's it. As a general rule, we want to make sure that a new model only depends on
[`PreTrainedModel`]. The important functionalities that are automatically provided to every new
model are [`~PreTrainedModel.from_pretrained`] and
[`~PreTrainedModel.save_pretrained`], which are used for serialization and deserialization. All of the
other important functionalities, such as `BrandNewBertModel.forward` should be completely defined in the new
`modeling_brand_new_bert.py` script. Next, we want to make sure that a model with a specific head layer, such as
`BrandNewBertForMaskedLM` does not inherit from `BrandNewBertModel`, but rather uses `BrandNewBertModel`
as a component that can be called in its forward pass to keep the level of abstraction low. Every new model requires a
configuration class, called `BrandNewBertConfig`.


key: _node_content
string_value:
{"id_": "bc62741c-9e8f-4154-b2af-eb946bdb517c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "73b2207f-dd9b-495a-8965-7ce9e62e2203", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "599d4cb4920a032486754f52278809d177cab7546e7ccd0005e7afb96865be20", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c455606e-8d49-4027-a6af-38f51d9f99c0", "node_type": "1", "metadata": {}, "hash": "db2e62acb5a2af71f77dce749b331e11674c5a3f6f81328e01718c3a19140510", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2886, "end_char_idx": 6173, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With this in mind, let's go a bit deeper into the general library design.

### Overview of models

To successfully add a model, it is important to understand the interaction between your model and its config,
[`PreTrainedModel`], and [`PretrainedConfig`]. For exemplary purposes, we will
call the model to be added to 🤗 Transformers `BrandNewBert`.

Let's take a look:

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers_overview.png"/>

As you can see, we do make use of inheritance in 🤗 Transformers, but we keep the level of abstraction to an absolute
minimum. There are never more than two levels of abstraction for any model in the library. `BrandNewBertModel`
inherits from `BrandNewBertPreTrainedModel` which in turn inherits from [`PreTrainedModel`] and
that's it. As a general rule, we want to make sure that a new model only depends on
[`PreTrainedModel`]. The important functionalities that are automatically provided to every new
model are [`~PreTrainedModel.from_pretrained`] and
[`~PreTrainedModel.save_pretrained`], which are used for serialization and deserialization. All of the
other important functionalities, such as `BrandNewBertModel.forward` should be completely defined in the new
`modeling_brand_new_bert.py` script. Next, we want to make sure that a model with a specific head layer, such as
`BrandNewBertForMaskedLM` does not inherit from `BrandNewBertModel`, but rather uses `BrandNewBertModel`
as a component that can be called in its forward pass to keep the level of abstraction low. Every new model requires a
configuration class, called `BrandNewBertConfig`. This configuration is always stored as an attribute in
[`PreTrainedModel`], and thus can be accessed via the `config` attribute for all classes
inheriting from `BrandNewBertPreTrainedModel`:

```python
model = BrandNewBertModel.from_pretrained("brandy/brand_new_bert")
model.config  # model has access to its config
```

Similar to the model, the configuration inherits basic serialization and deserialization functionalities from
[`PretrainedConfig`]. Note that the configuration and the model are always serialized into two
different formats - the model to a *pytorch_model.bin* file and the configuration to a *config.json* file. Calling
the model's [`~PreTrainedModel.save_pretrained`] will automatically call
the config's [`~PretrainedConfig.save_pretrained`], so that both model and configuration are saved.


### Code style

When coding your new model, keep in mind that Transformers is an opinionated library and we have a few quirks of our
own regarding how code should be written :-)

1. The forward pass of your model should be fully written in the modeling file while being fully independent of other
   models in the library. If you want to reuse a block from another model, copy the code and paste it with a
   `# Copied from` comment on top (see [here](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/models/roberta/modeling_roberta.py#L160)
   for a good example and [there](pr_checks#check-copies) for more documentation on Copied from). 
2. The code should be fully understandable, even by a non-native English speaker. This means you should pick
   descriptive variable names and avoid abbreviations.


key: _node_content
string_value:
{"id_": "c455606e-8d49-4027-a6af-38f51d9f99c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc62741c-9e8f-4154-b2af-eb946bdb517c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "8dffee4b0209628a0344fbbe9c6e72d3c0e2fefa62190277169a323e00892b15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3545bd2b-a6a4-4f2c-91cb-f38e076ca687", "node_type": "1", "metadata": {}, "hash": "aebd6107d9b984589933cf8bb5c321d05161a68552ef15f02410b8268f3e1539", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4528, "end_char_idx": 7719, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This configuration is always stored as an attribute in
[`PreTrainedModel`], and thus can be accessed via the `config` attribute for all classes
inheriting from `BrandNewBertPreTrainedModel`:

```python
model = BrandNewBertModel.from_pretrained("brandy/brand_new_bert")
model.config  # model has access to its config
```

Similar to the model, the configuration inherits basic serialization and deserialization functionalities from
[`PretrainedConfig`]. Note that the configuration and the model are always serialized into two
different formats - the model to a *pytorch_model.bin* file and the configuration to a *config.json* file. Calling
the model's [`~PreTrainedModel.save_pretrained`] will automatically call
the config's [`~PretrainedConfig.save_pretrained`], so that both model and configuration are saved.


### Code style

When coding your new model, keep in mind that Transformers is an opinionated library and we have a few quirks of our
own regarding how code should be written :-)

1. The forward pass of your model should be fully written in the modeling file while being fully independent of other
   models in the library. If you want to reuse a block from another model, copy the code and paste it with a
   `# Copied from` comment on top (see [here](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/models/roberta/modeling_roberta.py#L160)
   for a good example and [there](pr_checks#check-copies) for more documentation on Copied from). 
2. The code should be fully understandable, even by a non-native English speaker. This means you should pick
   descriptive variable names and avoid abbreviations. As an example, `activation` is preferred to `act`.
   One-letter variable names are strongly discouraged unless it's an index in a for loop.
3. More generally we prefer longer explicit code to short magical one.
4. Avoid subclassing `nn.Sequential` in PyTorch but subclass `nn.Module` and write the forward pass, so that anyone
   using your code can quickly debug it by adding print statements or breaking points.
5. Your function signature should be type-annotated. For the rest, good variable names are way more readable and
   understandable than type annotations.

### Overview of tokenizers

Not quite ready yet :-( This section will be added soon!

## Step-by-step recipe to add a model to 🤗 Transformers

Everyone has different preferences of how to port a model so it can be very helpful for you to take a look at summaries
of how other contributors ported models to Hugging Face. Here is a list of community blog posts on how to port a model:

1. [Porting GPT2 Model](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28) by [Thomas](https://huggingface.co/thomwolf)
2. [Porting WMT19 MT Model](https://huggingface.co/blog/porting-fsmt) by [Stas](https://huggingface.co/stas)

From experience, we can tell you that the most important things to keep in mind when adding a model are:

-  Don't reinvent the wheel! Most parts of the code you will add for the new 🤗 Transformers model already exist
  somewhere in 🤗 Transformers. Take some time to find similar, already existing models and tokenizers you can copy
  from.


key: _node_content
string_value:
{"id_": "3545bd2b-a6a4-4f2c-91cb-f38e076ca687", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c455606e-8d49-4027-a6af-38f51d9f99c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "efcadc0aba4f766d6b9d518f95092c142e69d9a44ef09a72cea5ef5c91eb86cb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1dc8342f-c992-4886-b144-61cae4f602a7", "node_type": "1", "metadata": {}, "hash": "c236486c6ce9da7fa02c53e36b0c14b99062c69816a971f539be47e6c58dc569", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6092, "end_char_idx": 8662, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means you should pick
   descriptive variable names and avoid abbreviations. As an example, `activation` is preferred to `act`.
   One-letter variable names are strongly discouraged unless it's an index in a for loop.
3. More generally we prefer longer explicit code to short magical one.
4. Avoid subclassing `nn.Sequential` in PyTorch but subclass `nn.Module` and write the forward pass, so that anyone
   using your code can quickly debug it by adding print statements or breaking points.
5. Your function signature should be type-annotated. For the rest, good variable names are way more readable and
   understandable than type annotations.

### Overview of tokenizers

Not quite ready yet :-( This section will be added soon!

## Step-by-step recipe to add a model to 🤗 Transformers

Everyone has different preferences of how to port a model so it can be very helpful for you to take a look at summaries
of how other contributors ported models to Hugging Face. Here is a list of community blog posts on how to port a model:

1. [Porting GPT2 Model](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28) by [Thomas](https://huggingface.co/thomwolf)
2. [Porting WMT19 MT Model](https://huggingface.co/blog/porting-fsmt) by [Stas](https://huggingface.co/stas)

From experience, we can tell you that the most important things to keep in mind when adding a model are:

-  Don't reinvent the wheel! Most parts of the code you will add for the new 🤗 Transformers model already exist
  somewhere in 🤗 Transformers. Take some time to find similar, already existing models and tokenizers you can copy
  from. [grep](https://www.gnu.org/software/grep/) and [rg](https://github.com/BurntSushi/ripgrep) are your
  friends. Note that it might very well happen that your model's tokenizer is based on one model implementation, and
  your model's modeling code on another one. *E.g.* FSMT's modeling code is based on BART, while FSMT's tokenizer code
  is based on XLM.
-  It's more of an engineering challenge than a scientific challenge. You should spend more time creating an
  efficient debugging environment rather than trying to understand all theoretical aspects of the model in the paper.
-  Ask for help, when you're stuck! Models are the core component of 🤗 Transformers so we at Hugging Face are more
  than happy to help you at every step to add your model. Don't hesitate to ask if you notice you are not making
  progress.

In the following, we try to give you a general recipe that we found most useful when porting a model to 🤗 Transformers.


key: _node_content
string_value:
{"id_": "1dc8342f-c992-4886-b144-61cae4f602a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3545bd2b-a6a4-4f2c-91cb-f38e076ca687", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "85ee1b28729c1ab320a1c6423134bb1cad525cd2b58c01dbb959dcc684614600", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a629fb84-ba84-40e9-8d04-35ea93c16137", "node_type": "1", "metadata": {}, "hash": "af9de8a25317467879b8e1cc32a798ce305c8c10626afd614c2ab9f35bb60a8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7131, "end_char_idx": 10359, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Porting GPT2 Model](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28) by [Thomas](https://huggingface.co/thomwolf)
2. [Porting WMT19 MT Model](https://huggingface.co/blog/porting-fsmt) by [Stas](https://huggingface.co/stas)

From experience, we can tell you that the most important things to keep in mind when adding a model are:

-  Don't reinvent the wheel! Most parts of the code you will add for the new 🤗 Transformers model already exist
  somewhere in 🤗 Transformers. Take some time to find similar, already existing models and tokenizers you can copy
  from. [grep](https://www.gnu.org/software/grep/) and [rg](https://github.com/BurntSushi/ripgrep) are your
  friends. Note that it might very well happen that your model's tokenizer is based on one model implementation, and
  your model's modeling code on another one. *E.g.* FSMT's modeling code is based on BART, while FSMT's tokenizer code
  is based on XLM.
-  It's more of an engineering challenge than a scientific challenge. You should spend more time creating an
  efficient debugging environment rather than trying to understand all theoretical aspects of the model in the paper.
-  Ask for help, when you're stuck! Models are the core component of 🤗 Transformers so we at Hugging Face are more
  than happy to help you at every step to add your model. Don't hesitate to ask if you notice you are not making
  progress.

In the following, we try to give you a general recipe that we found most useful when porting a model to 🤗 Transformers.

The following list is a summary of everything that has to be done to add a model and can be used by you as a To-Do
List:

☐ (Optional) Understood the model's theoretical aspects<br>
☐ Prepared 🤗 Transformers dev environment<br>
☐ Set up debugging environment of the original repository<br>
☐ Created script that successfully runs the `forward()` pass using the original repository and checkpoint<br>
☐ Successfully added the model skeleton to 🤗 Transformers<br>
☐ Successfully converted original checkpoint to 🤗 Transformers checkpoint<br>
☐ Successfully ran `forward()` pass in 🤗 Transformers that gives identical output to original checkpoint<br>
☐ Finished model tests in 🤗 Transformers<br>
☐ Successfully added tokenizer in 🤗 Transformers<br>
☐ Run end-to-end integration tests<br>
☐ Finished docs<br>
☐ Uploaded model weights to the Hub<br>
☐ Submitted the pull request<br>
☐ (Optional) Added a demo notebook

To begin with, we usually recommend starting by getting a good theoretical understanding of `BrandNewBert`. However,
if you prefer to understand the theoretical aspects of the model *on-the-job*, then it is totally fine to directly dive
into the `BrandNewBert`'s code-base. This option might suit you better if your engineering skills are better than
your theoretical skill, if you have trouble understanding `BrandNewBert`'s paper, or if you just enjoy programming
much more than reading scientific papers.

### 1. (Optional) Theoretical aspects of BrandNewBert

You should take some time to read *BrandNewBert's* paper, if such descriptive work exists. There might be large
sections of the paper that are difficult to understand. If this is the case, this is fine - don't worry!


key: _node_content
string_value:
{"id_": "a629fb84-ba84-40e9-8d04-35ea93c16137", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1dc8342f-c992-4886-b144-61cae4f602a7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "6353f44da589d0c2fbfcb84e7c2ee7a2c78ac1d05b2759e14e683f7178d08244", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6de27db9-e1d3-49d1-ad74-0ebe4707ae81", "node_type": "1", "metadata": {}, "hash": "948af9405745c9f24d617e415809f2cfbd55ec33e9c9e52eac5b8fb3add1476d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8664, "end_char_idx": 11945, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The following list is a summary of everything that has to be done to add a model and can be used by you as a To-Do
List:

☐ (Optional) Understood the model's theoretical aspects<br>
☐ Prepared 🤗 Transformers dev environment<br>
☐ Set up debugging environment of the original repository<br>
☐ Created script that successfully runs the `forward()` pass using the original repository and checkpoint<br>
☐ Successfully added the model skeleton to 🤗 Transformers<br>
☐ Successfully converted original checkpoint to 🤗 Transformers checkpoint<br>
☐ Successfully ran `forward()` pass in 🤗 Transformers that gives identical output to original checkpoint<br>
☐ Finished model tests in 🤗 Transformers<br>
☐ Successfully added tokenizer in 🤗 Transformers<br>
☐ Run end-to-end integration tests<br>
☐ Finished docs<br>
☐ Uploaded model weights to the Hub<br>
☐ Submitted the pull request<br>
☐ (Optional) Added a demo notebook

To begin with, we usually recommend starting by getting a good theoretical understanding of `BrandNewBert`. However,
if you prefer to understand the theoretical aspects of the model *on-the-job*, then it is totally fine to directly dive
into the `BrandNewBert`'s code-base. This option might suit you better if your engineering skills are better than
your theoretical skill, if you have trouble understanding `BrandNewBert`'s paper, or if you just enjoy programming
much more than reading scientific papers.

### 1. (Optional) Theoretical aspects of BrandNewBert

You should take some time to read *BrandNewBert's* paper, if such descriptive work exists. There might be large
sections of the paper that are difficult to understand. If this is the case, this is fine - don't worry! The goal is
not to get a deep theoretical understanding of the paper, but to extract the necessary information required to
effectively re-implement the model in 🤗 Transformers. That being said, you don't have to spend too much time on the
theoretical aspects, but rather focus on the practical ones, namely:

-  What type of model is *brand_new_bert*? BERT-like encoder-only model? GPT2-like decoder-only model? BART-like
  encoder-decoder model? Look at the [model_summary](model_summary) if you're not familiar with the differences between those.
-  What are the applications of *brand_new_bert*? Text classification? Text generation? Seq2Seq tasks, *e.g.,*
  summarization?
-  What is the novel feature of the model that makes it different from BERT/GPT-2/BART?
-  Which of the already existing [🤗 Transformers models](https://huggingface.co/transformers/#contents) is most
  similar to *brand_new_bert*?
-  What type of tokenizer is used? A sentencepiece tokenizer? Word piece tokenizer? Is it the same tokenizer as used
  for BERT or BART?

After you feel like you have gotten a good overview of the architecture of the model, you might want to write to the
Hugging Face team with any questions you might have. This might include questions regarding the model's architecture,
its attention layer, etc. We will be more than happy to help you.

### 2. Next prepare your environment

1. Fork the [repository](https://github.com/huggingface/transformers) by clicking on the ‘Fork' button on the
   repository's page. This creates a copy of the code under your GitHub user account.

2.


key: _node_content
string_value:
{"id_": "6de27db9-e1d3-49d1-ad74-0ebe4707ae81", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a629fb84-ba84-40e9-8d04-35ea93c16137", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "2fac60ef1bb3fbc91fe99cde393329444534c62e92f21744733b4bdb802a637d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "89d60114-8a6b-45a1-b984-d2e753789649", "node_type": "1", "metadata": {}, "hash": "9aaf36e639c7816d40b8eef8fdb0d0c60da77962cd551f7e49fd55fde9a094c1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10360, "end_char_idx": 13466, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The goal is
not to get a deep theoretical understanding of the paper, but to extract the necessary information required to
effectively re-implement the model in 🤗 Transformers. That being said, you don't have to spend too much time on the
theoretical aspects, but rather focus on the practical ones, namely:

-  What type of model is *brand_new_bert*? BERT-like encoder-only model? GPT2-like decoder-only model? BART-like
  encoder-decoder model? Look at the [model_summary](model_summary) if you're not familiar with the differences between those.
-  What are the applications of *brand_new_bert*? Text classification? Text generation? Seq2Seq tasks, *e.g.,*
  summarization?
-  What is the novel feature of the model that makes it different from BERT/GPT-2/BART?
-  Which of the already existing [🤗 Transformers models](https://huggingface.co/transformers/#contents) is most
  similar to *brand_new_bert*?
-  What type of tokenizer is used? A sentencepiece tokenizer? Word piece tokenizer? Is it the same tokenizer as used
  for BERT or BART?

After you feel like you have gotten a good overview of the architecture of the model, you might want to write to the
Hugging Face team with any questions you might have. This might include questions regarding the model's architecture,
its attention layer, etc. We will be more than happy to help you.

### 2. Next prepare your environment

1. Fork the [repository](https://github.com/huggingface/transformers) by clicking on the ‘Fork' button on the
   repository's page. This creates a copy of the code under your GitHub user account.

2. Clone your `transformers` fork to your local disk, and add the base repository as a remote:

   ```bash
   git clone https://github.com/[your Github handle]/transformers.git
   cd transformers
   git remote add upstream https://github.com/huggingface/transformers.git
   ```

3. Set up a development environment, for instance by running the following command:

   ```bash
   python -m venv .env
   source .env/bin/activate
   pip install -e ".[dev]"
   ```

   Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a
   failure with this command. If that's the case make sure to install the Deep Learning framework you are working with
   (PyTorch, TensorFlow and/or Flax) then do:

   ```bash
   pip install -e ".[quality]"
   ```

   which should be enough for most use cases. You can then return to the parent directory

   ```bash
   cd ..
   ```

4. We recommend adding the PyTorch version of *brand_new_bert* to Transformers. To install PyTorch, please follow the
   instructions on https://pytorch.org/get-started/locally/.

   **Note:** You don't need to have CUDA installed. Making the new model work on CPU is sufficient.

5. To port *brand_new_bert*, you will also need access to its original repository:

   ```bash
   git clone https://github.com/org_that_created_brand_new_bert_org/brand_new_bert.git
   cd brand_new_bert
   pip install -e .
   ```

Now you have set up a development environment to port *brand_new_bert* to 🤗 Transformers.

### 3.-4.


key: _node_content
string_value:
{"id_": "89d60114-8a6b-45a1-b984-d2e753789649", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6de27db9-e1d3-49d1-ad74-0ebe4707ae81", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "5a8a41d87fc34e6c79f100d3bf25cb9aacd7079f54e017ab85f663bb37975914", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1bfe1c74-4ffd-46eb-9522-0d60e713b548", "node_type": "1", "metadata": {}, "hash": "0349d8d3844d8364eac2e77a6c75e52a4af6bbcbb068e8172aa1b0e6fc1b9d54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11943, "end_char_idx": 15221, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. Clone your `transformers` fork to your local disk, and add the base repository as a remote:

   ```bash
   git clone https://github.com/[your Github handle]/transformers.git
   cd transformers
   git remote add upstream https://github.com/huggingface/transformers.git
   ```

3. Set up a development environment, for instance by running the following command:

   ```bash
   python -m venv .env
   source .env/bin/activate
   pip install -e ".[dev]"
   ```

   Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a
   failure with this command. If that's the case make sure to install the Deep Learning framework you are working with
   (PyTorch, TensorFlow and/or Flax) then do:

   ```bash
   pip install -e ".[quality]"
   ```

   which should be enough for most use cases. You can then return to the parent directory

   ```bash
   cd ..
   ```

4. We recommend adding the PyTorch version of *brand_new_bert* to Transformers. To install PyTorch, please follow the
   instructions on https://pytorch.org/get-started/locally/.

   **Note:** You don't need to have CUDA installed. Making the new model work on CPU is sufficient.

5. To port *brand_new_bert*, you will also need access to its original repository:

   ```bash
   git clone https://github.com/org_that_created_brand_new_bert_org/brand_new_bert.git
   cd brand_new_bert
   pip install -e .
   ```

Now you have set up a development environment to port *brand_new_bert* to 🤗 Transformers.

### 3.-4. Run a pretrained checkpoint using the original repository

At first, you will work on the original *brand_new_bert* repository. Often, the original implementation is very
“researchy”. Meaning that documentation might be lacking and the code can be difficult to understand. But this should
be exactly your motivation to reimplement *brand_new_bert*. At Hugging Face, one of our main goals is to *make people
stand on the shoulders of giants* which translates here very well into taking a working model and rewriting it to make
it as **accessible, user-friendly, and beautiful** as possible. This is the number-one motivation to re-implement
models into 🤗 Transformers - trying to make complex new NLP technology accessible to **everybody**.

You should start thereby by diving into the original repository.

Successfully running the official pretrained model in the original repository is often **the most difficult** step.
From our experience, it is very important to spend some time getting familiar with the original code-base. You need to
figure out the following:

- Where to find the pretrained weights?
- How to load the pretrained weights into the corresponding model?
- How to run the tokenizer independently from the model?
- Trace one forward pass so that you know which classes and functions are required for a simple forward pass. Usually,
  you only have to reimplement those functions.
- Be able to locate the important components of the model: Where is the model's class? Are there model sub-classes,
  *e.g.* EncoderModel, DecoderModel? Where is the self-attention layer? Are there multiple different attention layers,
  *e.g.* *self-attention*, *cross-attention*...?
- How can you debug the model in the original environment of the repo?


key: _node_content
string_value:
{"id_": "1bfe1c74-4ffd-46eb-9522-0d60e713b548", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "89d60114-8a6b-45a1-b984-d2e753789649", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "6780076fc18998b851cc70a5773957296d8892ca2057df54effde0563728ab6a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "829d3b24-073b-463c-bf5c-d964aa838209", "node_type": "1", "metadata": {}, "hash": "8b70049ff604f8f5d1e707e669783f73920110b664ecfbd0eb610b5e53a2b950", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13457, "end_char_idx": 16973, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### 3.-4. Run a pretrained checkpoint using the original repository

At first, you will work on the original *brand_new_bert* repository. Often, the original implementation is very
“researchy”. Meaning that documentation might be lacking and the code can be difficult to understand. But this should
be exactly your motivation to reimplement *brand_new_bert*. At Hugging Face, one of our main goals is to *make people
stand on the shoulders of giants* which translates here very well into taking a working model and rewriting it to make
it as **accessible, user-friendly, and beautiful** as possible. This is the number-one motivation to re-implement
models into 🤗 Transformers - trying to make complex new NLP technology accessible to **everybody**.

You should start thereby by diving into the original repository.

Successfully running the official pretrained model in the original repository is often **the most difficult** step.
From our experience, it is very important to spend some time getting familiar with the original code-base. You need to
figure out the following:

- Where to find the pretrained weights?
- How to load the pretrained weights into the corresponding model?
- How to run the tokenizer independently from the model?
- Trace one forward pass so that you know which classes and functions are required for a simple forward pass. Usually,
  you only have to reimplement those functions.
- Be able to locate the important components of the model: Where is the model's class? Are there model sub-classes,
  *e.g.* EncoderModel, DecoderModel? Where is the self-attention layer? Are there multiple different attention layers,
  *e.g.* *self-attention*, *cross-attention*...?
- How can you debug the model in the original environment of the repo? Do you have to add *print* statements, can you
  work with an interactive debugger like *ipdb*, or should you use an efficient IDE to debug the model, like PyCharm?

It is very important that before you start the porting process, you can **efficiently** debug code in the original
repository! Also, remember that you are working with an open-source library, so do not hesitate to open an issue, or
even a pull request in the original repository. The maintainers of this repository are most likely very happy about
someone looking into their code!

At this point, it is really up to you which debugging environment and strategy you prefer to use to debug the original
model. We strongly advise against setting up a costly GPU environment, but simply work on a CPU both when starting to
dive into the original repository and also when starting to write the 🤗 Transformers implementation of the model. Only
at the very end, when the model has already been successfully ported to 🤗 Transformers, one should verify that the
model also works as expected on GPU.

In general, there are two possible debugging environments for running the original model

-  [Jupyter notebooks](https://jupyter.org/) / [google colab](https://colab.research.google.com/notebooks/intro.ipynb)
-  Local python scripts.

Jupyter notebooks have the advantage that they allow for cell-by-cell execution which can be helpful to better split
logical components from one another and to have faster debugging cycles as intermediate results can be stored. Also,
notebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging
Face team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them.


key: _node_content
string_value:
{"id_": "829d3b24-073b-463c-bf5c-d964aa838209", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1bfe1c74-4ffd-46eb-9522-0d60e713b548", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "dd05effb5d9216329253572d6d1cffec7b63f0f9f895b582d6503f0b47531030", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1f5ae571-12b0-42a3-8e53-703e7da54346", "node_type": "1", "metadata": {}, "hash": "ae1f10527656926adfd0bd2fab62ba4aa1cd3f58df2f7edfea75b640d7b87db6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15151, "end_char_idx": 18407, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- How can you debug the model in the original environment of the repo? Do you have to add *print* statements, can you
  work with an interactive debugger like *ipdb*, or should you use an efficient IDE to debug the model, like PyCharm?

It is very important that before you start the porting process, you can **efficiently** debug code in the original
repository! Also, remember that you are working with an open-source library, so do not hesitate to open an issue, or
even a pull request in the original repository. The maintainers of this repository are most likely very happy about
someone looking into their code!

At this point, it is really up to you which debugging environment and strategy you prefer to use to debug the original
model. We strongly advise against setting up a costly GPU environment, but simply work on a CPU both when starting to
dive into the original repository and also when starting to write the 🤗 Transformers implementation of the model. Only
at the very end, when the model has already been successfully ported to 🤗 Transformers, one should verify that the
model also works as expected on GPU.

In general, there are two possible debugging environments for running the original model

-  [Jupyter notebooks](https://jupyter.org/) / [google colab](https://colab.research.google.com/notebooks/intro.ipynb)
-  Local python scripts.

Jupyter notebooks have the advantage that they allow for cell-by-cell execution which can be helpful to better split
logical components from one another and to have faster debugging cycles as intermediate results can be stored. Also,
notebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging
Face team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them.

The obvious disadvantage of Jupyter notebooks is that if you are not used to working with them you will have to spend
some time adjusting to the new programming environment and you might not be able to use your known debugging tools
anymore, like `ipdb`.

For each code-base, a good first step is always to load a **small** pretrained checkpoint and to be able to reproduce a
single forward pass using a dummy integer vector of input IDs as an input. Such a script could look like this (in
pseudocode):

```python
model = BrandNewBertModel.load_pretrained_checkpoint("/path/to/checkpoint/")
input_ids = [0, 4, 5, 2, 3, 7, 9]  # vector of input ids
original_output = model.predict(input_ids)
```

Next, regarding the debugging strategy, there are generally a few from which to choose from:

- Decompose the original model into many small testable components and run a forward pass on each of those for
  verification
- Decompose the original model only into the original *tokenizer* and the original *model*, run a forward pass on
  those, and use intermediate print statements or breakpoints for verification

Again, it is up to you which strategy to choose. Often, one or the other is advantageous depending on the original code
base.

If the original code-base allows you to decompose the model into smaller sub-components, *e.g.* if the original
code-base can easily be run in eager mode, it is usually worth the effort to do so.


key: _node_content
string_value:
{"id_": "1f5ae571-12b0-42a3-8e53-703e7da54346", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "829d3b24-073b-463c-bf5c-d964aa838209", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "321653e0f8b1395e19610df2d871d32b4a7d5c6a4488c856923333ceffd663b2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6af0946d-bdbf-4ddf-9663-891222978fd0", "node_type": "1", "metadata": {}, "hash": "a736c2ca4d0fdfeec83a5fc5a01243831daeabceef8f2c8e58b908fc4bdce857", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16742, "end_char_idx": 20335, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Also,
notebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging
Face team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them.

The obvious disadvantage of Jupyter notebooks is that if you are not used to working with them you will have to spend
some time adjusting to the new programming environment and you might not be able to use your known debugging tools
anymore, like `ipdb`.

For each code-base, a good first step is always to load a **small** pretrained checkpoint and to be able to reproduce a
single forward pass using a dummy integer vector of input IDs as an input. Such a script could look like this (in
pseudocode):

```python
model = BrandNewBertModel.load_pretrained_checkpoint("/path/to/checkpoint/")
input_ids = [0, 4, 5, 2, 3, 7, 9]  # vector of input ids
original_output = model.predict(input_ids)
```

Next, regarding the debugging strategy, there are generally a few from which to choose from:

- Decompose the original model into many small testable components and run a forward pass on each of those for
  verification
- Decompose the original model only into the original *tokenizer* and the original *model*, run a forward pass on
  those, and use intermediate print statements or breakpoints for verification

Again, it is up to you which strategy to choose. Often, one or the other is advantageous depending on the original code
base.

If the original code-base allows you to decompose the model into smaller sub-components, *e.g.* if the original
code-base can easily be run in eager mode, it is usually worth the effort to do so. There are some important advantages
to taking the more difficult road in the beginning:

- at a later stage when comparing the original model to the Hugging Face implementation, you can verify automatically
  for each component individually that the corresponding component of the 🤗 Transformers implementation matches instead
  of relying on visual comparison via print statements
- it can give you some rope to decompose the big problem of porting a model into smaller problems of just porting
  individual components and thus structure your work better
- separating the model into logical meaningful components will help you to get a better overview of the model's design
  and thus to better understand the model
- at a later stage those component-by-component tests help you to ensure that no regression occurs as you continue
  changing your code

[Lysandre's](https://gist.github.com/LysandreJik/db4c948f6b4483960de5cbac598ad4ed) integration checks for ELECTRA
gives a nice example of how this can be done.

However, if the original code-base is very complex or only allows intermediate components to be run in a compiled mode,
it might be too time-consuming or even impossible to separate the model into smaller testable sub-components. A good
example is [T5's MeshTensorFlow](https://github.com/tensorflow/mesh/tree/master/mesh_tensorflow) library which is
very complex and does not offer a simple way to decompose the model into its sub-components. For such libraries, one
often relies on verifying print statements.

No matter which strategy you choose, the recommended procedure is often the same that you should start to debug the
starting layers first and the ending layers last.

It is recommended that you retrieve the output, either by print statements or sub-component functions, of the following
layers in the following order:

1. Retrieve the input IDs passed to the model
2. Retrieve the word embeddings
3.


key: _node_content
string_value:
{"id_": "6af0946d-bdbf-4ddf-9663-891222978fd0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1f5ae571-12b0-42a3-8e53-703e7da54346", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "cde417571e1b6facd61edf08699322359edbe83ab4a20ee473e59733c8733024", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "58224085-0fd6-4122-a327-878cce67a24f", "node_type": "1", "metadata": {}, "hash": "fb0c9573da64959e8a525f2ee61e1e6326cf6c6c04dc8cc78370563f589f92ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19423, "end_char_idx": 22179, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, if the original code-base is very complex or only allows intermediate components to be run in a compiled mode,
it might be too time-consuming or even impossible to separate the model into smaller testable sub-components. A good
example is [T5's MeshTensorFlow](https://github.com/tensorflow/mesh/tree/master/mesh_tensorflow) library which is
very complex and does not offer a simple way to decompose the model into its sub-components. For such libraries, one
often relies on verifying print statements.

No matter which strategy you choose, the recommended procedure is often the same that you should start to debug the
starting layers first and the ending layers last.

It is recommended that you retrieve the output, either by print statements or sub-component functions, of the following
layers in the following order:

1. Retrieve the input IDs passed to the model
2. Retrieve the word embeddings
3. Retrieve the input of the first Transformer layer
4. Retrieve the output of the first Transformer layer
5. Retrieve the output of the following n - 1 Transformer layers
6. Retrieve the output of the whole BrandNewBert Model

Input IDs should thereby consists of an array of integers, *e.g.* `input_ids = [0, 4, 4, 3, 2, 4, 1, 7, 19]`

The outputs of the following layers often consist of multi-dimensional float arrays and can look like this:

```
[[
 [-0.1465, -0.6501,  0.1993,  ...,  0.1451,  0.3430,  0.6024],
 [-0.4417, -0.5920,  0.3450,  ..., -0.3062,  0.6182,  0.7132],
 [-0.5009, -0.7122,  0.4548,  ..., -0.3662,  0.6091,  0.7648],
 ...,
 [-0.5613, -0.6332,  0.4324,  ..., -0.3792,  0.7372,  0.9288],
 [-0.5416, -0.6345,  0.4180,  ..., -0.3564,  0.6992,  0.9191],
 [-0.5334, -0.6403,  0.4271,  ..., -0.3339,  0.6533,  0.8694]]],
```

We expect that every model added to 🤗 Transformers passes a couple of integration tests, meaning that the original
model and the reimplemented version in 🤗 Transformers have to give the exact same output up to a precision of 0.001!
Since it is normal that the exact same model written in different libraries can give a slightly different output
depending on the library framework, we accept an error tolerance of 1e-3 (0.001). It is not enough if the model gives
nearly the same output, they have to be almost identical. Therefore, you will certainly compare the intermediate
outputs of the 🤗 Transformers version multiple times against the intermediate outputs of the original implementation of
*brand_new_bert* in which case an **efficient** debugging environment of the original repository is absolutely
important. Here is some advice to make your debugging environment as efficient as possible.

- Find the best way of debugging intermediate results. Is the original repository written in PyTorch?


key: _node_content
string_value:
{"id_": "58224085-0fd6-4122-a327-878cce67a24f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6af0946d-bdbf-4ddf-9663-891222978fd0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "2e8c025d4eebfd604ab599cbaedc37b9d2f3e011e92ce8113bad947ca548e503", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4a9d097-f592-44ed-b777-a7bc3c182647", "node_type": "1", "metadata": {}, "hash": "eaff508018d10e9ea1cd7cc466d9d0b30ca83da4f9f8d714ff6c7e3c3a855dde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21410, "end_char_idx": 24711, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since it is normal that the exact same model written in different libraries can give a slightly different output
depending on the library framework, we accept an error tolerance of 1e-3 (0.001). It is not enough if the model gives
nearly the same output, they have to be almost identical. Therefore, you will certainly compare the intermediate
outputs of the 🤗 Transformers version multiple times against the intermediate outputs of the original implementation of
*brand_new_bert* in which case an **efficient** debugging environment of the original repository is absolutely
important. Here is some advice to make your debugging environment as efficient as possible.

- Find the best way of debugging intermediate results. Is the original repository written in PyTorch? Then you should
  probably take the time to write a longer script that decomposes the original model into smaller sub-components to
  retrieve intermediate values. Is the original repository written in Tensorflow 1? Then you might have to rely on
  TensorFlow print operations like [tf.print](https://www.tensorflow.org/api_docs/python/tf/print) to output
  intermediate values. Is the original repository written in Jax? Then make sure that the model is **not jitted** when
  running the forward pass, *e.g.* check-out [this link](https://github.com/google/jax/issues/196).
- Use the smallest pretrained checkpoint you can find. The smaller the checkpoint, the faster your debug cycle
  becomes. It is not efficient if your pretrained model is so big that your forward pass takes more than 10 seconds.
  In case only very large checkpoints are available, it might make more sense to create a dummy model in the new
  environment with randomly initialized weights and save those weights for comparison with the 🤗 Transformers version
  of your model
- Make sure you are using the easiest way of calling a forward pass in the original repository. Ideally, you want to
  find the function in the original repository that **only** calls a single forward pass, *i.e.* that is often called
  `predict`, `evaluate`, `forward` or `__call__`. You don't want to debug a function that calls `forward`
  multiple times, *e.g.* to generate text, like `autoregressive_sample`, `generate`.
- Try to separate the tokenization from the model's *forward* pass. If the original repository shows examples where
  you have to input a string, then try to find out where in the forward call the string input is changed to input ids
  and start from this point. This might mean that you have to possibly write a small script yourself or change the
  original code so that you can directly input the ids instead of an input string.
- Make sure that the model in your debugging setup is **not** in training mode, which often causes the model to yield
  random outputs due to multiple dropout layers in the model. Make sure that the forward pass in your debugging
  environment is **deterministic** so that the dropout layers are not used. Or use *transformers.utils.set_seed*
  if the old and new implementations are in the same framework.

The following section gives you more specific details/tips on how you can do this for *brand_new_bert*.

### 5.-14. Port BrandNewBert to 🤗 Transformers

Next, you can finally start adding new code to 🤗 Transformers.


key: _node_content
string_value:
{"id_": "c4a9d097-f592-44ed-b777-a7bc3c182647", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "58224085-0fd6-4122-a327-878cce67a24f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "9ada35ed9444acbc9b74a97bd73c7dae6316f40cb8d39dc6d525b6f8665ce218", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "00f0b085-2bf6-4907-83ed-1d17ba913535", "node_type": "1", "metadata": {}, "hash": "e921c7fee3dbba93ca2ac34a7691c11fa78995341a54135017835230b585ecbe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22985, "end_char_idx": 26304, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In case only very large checkpoints are available, it might make more sense to create a dummy model in the new
  environment with randomly initialized weights and save those weights for comparison with the 🤗 Transformers version
  of your model
- Make sure you are using the easiest way of calling a forward pass in the original repository. Ideally, you want to
  find the function in the original repository that **only** calls a single forward pass, *i.e.* that is often called
  `predict`, `evaluate`, `forward` or `__call__`. You don't want to debug a function that calls `forward`
  multiple times, *e.g.* to generate text, like `autoregressive_sample`, `generate`.
- Try to separate the tokenization from the model's *forward* pass. If the original repository shows examples where
  you have to input a string, then try to find out where in the forward call the string input is changed to input ids
  and start from this point. This might mean that you have to possibly write a small script yourself or change the
  original code so that you can directly input the ids instead of an input string.
- Make sure that the model in your debugging setup is **not** in training mode, which often causes the model to yield
  random outputs due to multiple dropout layers in the model. Make sure that the forward pass in your debugging
  environment is **deterministic** so that the dropout layers are not used. Or use *transformers.utils.set_seed*
  if the old and new implementations are in the same framework.

The following section gives you more specific details/tips on how you can do this for *brand_new_bert*.

### 5.-14. Port BrandNewBert to 🤗 Transformers

Next, you can finally start adding new code to 🤗 Transformers. Go into the clone of your 🤗 Transformers' fork:

```bash
cd transformers
```

In the special case that you are adding a model whose architecture exactly matches the model architecture of an
existing model you only have to add a conversion script as described in [this section](#write-a-conversion-script).
In this case, you can just re-use the whole model architecture of the already existing model.

Otherwise, let's start generating a new model. We recommend using the following script to add a model starting from
an existing model:

```bash
transformers-cli add-new-model-like
```

You will be prompted with a questionnaire to fill in the basic information of your model.

**Open a Pull Request on the main huggingface/transformers repo**

Before starting to adapt the automatically generated code, now is the time to open a “Work in progress (WIP)” pull
request, *e.g.* “[WIP] Add *brand_new_bert*”, in 🤗 Transformers so that you and the Hugging Face team can work
side-by-side on integrating the model into 🤗 Transformers.

You should do the following:

1. Create a branch with a descriptive name from your main branch

   ```bash
   git checkout -b add_brand_new_bert
   ```

2. Commit the automatically generated code:

   ```bash
   git add .
   git commit
   ```

3. Fetch and rebase to current main

   ```bash
   git fetch upstream
   git rebase upstream/main
   ```

4. Push the changes to your account using:

   ```bash
   git push -u origin a-descriptive-name-for-my-changes
   ```

5. Once you are satisfied, go to the webpage of your fork on GitHub. Click on “Pull request”.


key: _node_content
string_value:
{"id_": "00f0b085-2bf6-4907-83ed-1d17ba913535", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4a9d097-f592-44ed-b777-a7bc3c182647", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "aed4a4fa678cc069f832a5ca2526338f94a8deb9021ac1c10d13f746c6e35a4a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c4a15df-6ac4-4f67-bedc-234a80500213", "node_type": "1", "metadata": {}, "hash": "20f149cb0d6b0f8977e675d2a7146ad47e9912ee7cc77612ac04c2e637a54a32", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24712, "end_char_idx": 28028, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Go into the clone of your 🤗 Transformers' fork:

```bash
cd transformers
```

In the special case that you are adding a model whose architecture exactly matches the model architecture of an
existing model you only have to add a conversion script as described in [this section](#write-a-conversion-script).
In this case, you can just re-use the whole model architecture of the already existing model.

Otherwise, let's start generating a new model. We recommend using the following script to add a model starting from
an existing model:

```bash
transformers-cli add-new-model-like
```

You will be prompted with a questionnaire to fill in the basic information of your model.

**Open a Pull Request on the main huggingface/transformers repo**

Before starting to adapt the automatically generated code, now is the time to open a “Work in progress (WIP)” pull
request, *e.g.* “[WIP] Add *brand_new_bert*”, in 🤗 Transformers so that you and the Hugging Face team can work
side-by-side on integrating the model into 🤗 Transformers.

You should do the following:

1. Create a branch with a descriptive name from your main branch

   ```bash
   git checkout -b add_brand_new_bert
   ```

2. Commit the automatically generated code:

   ```bash
   git add .
   git commit
   ```

3. Fetch and rebase to current main

   ```bash
   git fetch upstream
   git rebase upstream/main
   ```

4. Push the changes to your account using:

   ```bash
   git push -u origin a-descriptive-name-for-my-changes
   ```

5. Once you are satisfied, go to the webpage of your fork on GitHub. Click on “Pull request”. Make sure to add the
   GitHub handle of some members of the Hugging Face team as reviewers, so that the Hugging Face team gets notified for
   future changes.

6. Change the PR into a draft by clicking on “Convert to draft” on the right of the GitHub pull request web page.

In the following, whenever you have made some progress, don't forget to commit your work and push it to your account so
that it shows in the pull request. Additionally, you should make sure to update your work with the current main from
time to time by doing:

```bash
git fetch upstream
git merge upstream/main
```

In general, all questions you might have regarding the model or your implementation should be asked in your PR and
discussed/solved in the PR. This way, the Hugging Face team will always be notified when you are committing new code or
if you have a question. It is often very helpful to point the Hugging Face team to your added code so that the Hugging
Face team can efficiently understand your problem or question.

To do so, you can go to the “Files changed” tab where you see all of your changes, go to a line regarding which you
want to ask a question, and click on the “+” symbol to add a comment. Whenever a question or problem has been solved,
you can click on the “Resolve” button of the created comment.

In the same way, the Hugging Face team will open comments when reviewing your code. We recommend asking most questions
on GitHub on your PR. For some very general questions that are not very useful for the public, feel free to ping the
Hugging Face team by Slack or email.

**5. Adapt the generated models code for brand_new_bert**

At first, we will focus only on the model itself and not care about the tokenizer.


key: _node_content
string_value:
{"id_": "8c4a15df-6ac4-4f67-bedc-234a80500213", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "00f0b085-2bf6-4907-83ed-1d17ba913535", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "248c8d63a5f70a05125001442207460dd278df6b942fd7743725879f01aa69ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c9b03a1-dbc4-4a2d-9c98-bc6437882a7b", "node_type": "1", "metadata": {}, "hash": "8e2212f255b0b3d4ec2a1f39de3541255cf8cdbf9dfcde027eba571fa7784bd2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26305, "end_char_idx": 29599, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Make sure to add the
   GitHub handle of some members of the Hugging Face team as reviewers, so that the Hugging Face team gets notified for
   future changes.

6. Change the PR into a draft by clicking on “Convert to draft” on the right of the GitHub pull request web page.

In the following, whenever you have made some progress, don't forget to commit your work and push it to your account so
that it shows in the pull request. Additionally, you should make sure to update your work with the current main from
time to time by doing:

```bash
git fetch upstream
git merge upstream/main
```

In general, all questions you might have regarding the model or your implementation should be asked in your PR and
discussed/solved in the PR. This way, the Hugging Face team will always be notified when you are committing new code or
if you have a question. It is often very helpful to point the Hugging Face team to your added code so that the Hugging
Face team can efficiently understand your problem or question.

To do so, you can go to the “Files changed” tab where you see all of your changes, go to a line regarding which you
want to ask a question, and click on the “+” symbol to add a comment. Whenever a question or problem has been solved,
you can click on the “Resolve” button of the created comment.

In the same way, the Hugging Face team will open comments when reviewing your code. We recommend asking most questions
on GitHub on your PR. For some very general questions that are not very useful for the public, feel free to ping the
Hugging Face team by Slack or email.

**5. Adapt the generated models code for brand_new_bert**

At first, we will focus only on the model itself and not care about the tokenizer. All the relevant code should be
found in the generated files `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` and
`src/transformers/models/brand_new_bert/configuration_brand_new_bert.py`.

Now you can finally start coding :). The generated code in
`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` will either have the same architecture as BERT if
it's an encoder-only model or BART if it's an encoder-decoder model. At this point, you should remind yourself what
you've learned in the beginning about the theoretical aspects of the model: *How is the model different from BERT or
BART?*". Implement those changes which often means changing the *self-attention* layer, the order of the normalization
layer, etc… Again, it is often useful to look at the similar architecture of already existing models in Transformers to
get a better feeling of how your model should be implemented.

**Note** that at this point, you don't have to be very sure that your code is fully correct or clean. Rather, it is
advised to add a first *unclean*, copy-pasted version of the original code to
`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` until you feel like all the necessary code is
added. From our experience, it is much more efficient to quickly add a first version of the required code and
improve/correct the code iteratively with the conversion script as described in the next section. The only thing that
has to work at this point is that you can instantiate the 🤗 Transformers implementation of *brand_new_bert*, *i.e.


key: _node_content
string_value:
{"id_": "2c9b03a1-dbc4-4a2d-9c98-bc6437882a7b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c4a15df-6ac4-4f67-bedc-234a80500213", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "d5829f36c3c4008a64a785966f778ccadf56e69a2c6ff5e6ec209efd67957489", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a1205e86-fda5-49ee-8fca-d01d5a673433", "node_type": "1", "metadata": {}, "hash": "db18b816a8bb6f0bb1a4f3592d6a3b6c7ece174defcf84fdd6e00208b8fe390e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27892, "end_char_idx": 31126, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Adapt the generated models code for brand_new_bert**

At first, we will focus only on the model itself and not care about the tokenizer. All the relevant code should be
found in the generated files `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` and
`src/transformers/models/brand_new_bert/configuration_brand_new_bert.py`.

Now you can finally start coding :). The generated code in
`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` will either have the same architecture as BERT if
it's an encoder-only model or BART if it's an encoder-decoder model. At this point, you should remind yourself what
you've learned in the beginning about the theoretical aspects of the model: *How is the model different from BERT or
BART?*". Implement those changes which often means changing the *self-attention* layer, the order of the normalization
layer, etc… Again, it is often useful to look at the similar architecture of already existing models in Transformers to
get a better feeling of how your model should be implemented.

**Note** that at this point, you don't have to be very sure that your code is fully correct or clean. Rather, it is
advised to add a first *unclean*, copy-pasted version of the original code to
`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` until you feel like all the necessary code is
added. From our experience, it is much more efficient to quickly add a first version of the required code and
improve/correct the code iteratively with the conversion script as described in the next section. The only thing that
has to work at this point is that you can instantiate the 🤗 Transformers implementation of *brand_new_bert*, *i.e.* the
following command should work:

```python
from transformers import BrandNewBertModel, BrandNewBertConfig

model = BrandNewBertModel(BrandNewBertConfig())
```

The above command will create a model according to the default parameters as defined in `BrandNewBertConfig()` with
random weights, thus making sure that the `init()` methods of all components works.

Note that all random initialization should happen in the `_init_weights` method of your `BrandnewBertPreTrainedModel`
class. It should initialize all leaf modules depending on the variables of the config. Here is an example with the
BERT `_init_weights` method:

```py
def _init_weights(self, module):
    """Initialize the weights"""
    if isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()
    elif isinstance(module, nn.Embedding):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.padding_idx is not None:
            module.weight.data[module.padding_idx].zero_()
    elif isinstance(module, nn.LayerNorm):
        module.bias.data.zero_()
        module.weight.data.fill_(1.0)
```

You can have some more custom schemes if you need a special initialization for some modules. For instance, in
`Wav2Vec2ForPreTraining`, the last two linear layers need to have the initialization of the regular PyTorch `nn.Linear`
but all the other ones should use an initialization as above.


key: _node_content
string_value:
{"id_": "a1205e86-fda5-49ee-8fca-d01d5a673433", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c9b03a1-dbc4-4a2d-9c98-bc6437882a7b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "ae0114515928a2e185b6a6b2ff16e1a6e6a4449445949b82c72b5cfd86b5c0c4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7d815f37-73ac-4c71-be0c-e74396a2aa51", "node_type": "1", "metadata": {}, "hash": "6b72d901339824be5511fecce72586feb08fa401f616b29675add2a5ded0bf0e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29465, "end_char_idx": 32734, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The only thing that
has to work at this point is that you can instantiate the 🤗 Transformers implementation of *brand_new_bert*, *i.e.* the
following command should work:

```python
from transformers import BrandNewBertModel, BrandNewBertConfig

model = BrandNewBertModel(BrandNewBertConfig())
```

The above command will create a model according to the default parameters as defined in `BrandNewBertConfig()` with
random weights, thus making sure that the `init()` methods of all components works.

Note that all random initialization should happen in the `_init_weights` method of your `BrandnewBertPreTrainedModel`
class. It should initialize all leaf modules depending on the variables of the config. Here is an example with the
BERT `_init_weights` method:

```py
def _init_weights(self, module):
    """Initialize the weights"""
    if isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()
    elif isinstance(module, nn.Embedding):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.padding_idx is not None:
            module.weight.data[module.padding_idx].zero_()
    elif isinstance(module, nn.LayerNorm):
        module.bias.data.zero_()
        module.weight.data.fill_(1.0)
```

You can have some more custom schemes if you need a special initialization for some modules. For instance, in
`Wav2Vec2ForPreTraining`, the last two linear layers need to have the initialization of the regular PyTorch `nn.Linear`
but all the other ones should use an initialization as above. This is coded like this:

```py
def _init_weights(self, module):
    """Initialize the weights"""
    if isinstance(module, Wav2Vec2ForPreTraining):
        module.project_hid.reset_parameters()
        module.project_q.reset_parameters()
        module.project_hid._is_hf_initialized = True
        module.project_q._is_hf_initialized = True
    elif isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()
```

The `_is_hf_initialized` flag is internally used to make sure we only initialize a submodule once. By setting it to
`True` for `module.project_q` and `module.project_hid`, we make sure the custom initialization we did is not overridden later on,
the `_init_weights` function won't be applied to them.

**6. Write a conversion script**

Next, you should write a conversion script that lets you convert the checkpoint you used to debug *brand_new_bert* in
the original repository to a checkpoint compatible with your just created 🤗 Transformers implementation of
*brand_new_bert*. It is not advised to write the conversion script from scratch, but rather to look through already
existing conversion scripts in 🤗 Transformers for one that has been used to convert a similar model that was written in
the same framework as *brand_new_bert*. Usually, it is enough to copy an already existing conversion script and
slightly adapt it for your use case. Don't hesitate to ask the Hugging Face team to point you to a similar already
existing conversion script for your model.


key: _node_content
string_value:
{"id_": "7d815f37-73ac-4c71-be0c-e74396a2aa51", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a1205e86-fda5-49ee-8fca-d01d5a673433", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "51f094ff6ee5d474b9b26600a36717f2b47f9f47bf2f756cd54aeeea3652f962", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3643d16a-cb55-4f72-8d7a-4e207646c944", "node_type": "1", "metadata": {}, "hash": "2ccc8f19db1200e2188c86c7db5e40ba74075a88a1a2956a5fae250f51c6e58a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30928, "end_char_idx": 33917, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For instance, in
`Wav2Vec2ForPreTraining`, the last two linear layers need to have the initialization of the regular PyTorch `nn.Linear`
but all the other ones should use an initialization as above. This is coded like this:

```py
def _init_weights(self, module):
    """Initialize the weights"""
    if isinstance(module, Wav2Vec2ForPreTraining):
        module.project_hid.reset_parameters()
        module.project_q.reset_parameters()
        module.project_hid._is_hf_initialized = True
        module.project_q._is_hf_initialized = True
    elif isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()
```

The `_is_hf_initialized` flag is internally used to make sure we only initialize a submodule once. By setting it to
`True` for `module.project_q` and `module.project_hid`, we make sure the custom initialization we did is not overridden later on,
the `_init_weights` function won't be applied to them.

**6. Write a conversion script**

Next, you should write a conversion script that lets you convert the checkpoint you used to debug *brand_new_bert* in
the original repository to a checkpoint compatible with your just created 🤗 Transformers implementation of
*brand_new_bert*. It is not advised to write the conversion script from scratch, but rather to look through already
existing conversion scripts in 🤗 Transformers for one that has been used to convert a similar model that was written in
the same framework as *brand_new_bert*. Usually, it is enough to copy an already existing conversion script and
slightly adapt it for your use case. Don't hesitate to ask the Hugging Face team to point you to a similar already
existing conversion script for your model.

- If you are porting a model from TensorFlow to PyTorch, a good starting point might be BERT's conversion script [here](https://github.com/huggingface/transformers/blob/7acfa95afb8194f8f9c1f4d2c6028224dbed35a2/src/transformers/models/bert/modeling_bert.py#L91)
- If you are porting a model from PyTorch to PyTorch, a good starting point might be BART's conversion script [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/convert_bart_original_pytorch_checkpoint_to_pytorch.py)

In the following, we'll quickly explain how PyTorch models store layer weights and define layer names. In PyTorch, the
name of a layer is defined by the name of the class attribute you give the layer. Let's define a dummy model in
PyTorch, called `SimpleModel` as follows:

```python
from torch import nn


class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.dense = nn.Linear(10, 10)
        self.intermediate = nn.Linear(10, 10)
        self.layer_norm = nn.LayerNorm(10)
```

Now we can create an instance of this model definition which will fill all weights: `dense`, `intermediate`,
`layer_norm` with random weights.


key: _node_content
string_value:
{"id_": "3643d16a-cb55-4f72-8d7a-4e207646c944", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7d815f37-73ac-4c71-be0c-e74396a2aa51", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "50658d17d1e104ffb45ae39b0bdd7bd16bcfb99a8be36400a1e6f5a05a8ab61c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2e91d56-6c47-4f49-817d-ff81a4f44c12", "node_type": "1", "metadata": {}, "hash": "dcc459908fc9931a611151b70f769b6dc407c463e9f82a6a53b2d35720959c46", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32505, "end_char_idx": 34367, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Usually, it is enough to copy an already existing conversion script and
slightly adapt it for your use case. Don't hesitate to ask the Hugging Face team to point you to a similar already
existing conversion script for your model.

- If you are porting a model from TensorFlow to PyTorch, a good starting point might be BERT's conversion script [here](https://github.com/huggingface/transformers/blob/7acfa95afb8194f8f9c1f4d2c6028224dbed35a2/src/transformers/models/bert/modeling_bert.py#L91)
- If you are porting a model from PyTorch to PyTorch, a good starting point might be BART's conversion script [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/convert_bart_original_pytorch_checkpoint_to_pytorch.py)

In the following, we'll quickly explain how PyTorch models store layer weights and define layer names. In PyTorch, the
name of a layer is defined by the name of the class attribute you give the layer. Let's define a dummy model in
PyTorch, called `SimpleModel` as follows:

```python
from torch import nn


class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.dense = nn.Linear(10, 10)
        self.intermediate = nn.Linear(10, 10)
        self.layer_norm = nn.LayerNorm(10)
```

Now we can create an instance of this model definition which will fill all weights: `dense`, `intermediate`,
`layer_norm` with random weights. We can print the model to see its architecture

```python
model = SimpleModel()

print(model)
```

This will print out the following:

```
SimpleModel(
  (dense): Linear(in_features=10, out_features=10, bias=True)
  (intermediate): Linear(in_features=10, out_features=10, bias=True)
  (layer_norm): LayerNorm((10,), eps=1e-05, elementwise_affine=True)
)
```

We can see that the layer names are defined by the name of the class attribute in PyTorch.


key: _node_content
string_value:
{"id_": "d2e91d56-6c47-4f49-817d-ff81a4f44c12", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3643d16a-cb55-4f72-8d7a-4e207646c944", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "3beec97ae0ec67f8e1ebedb2404ffc24efe524b51848143cc43ca0ecd5f9907a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a006fa83-d534-444b-b48f-b0c1c04981bc", "node_type": "1", "metadata": {}, "hash": "85865ed2b2bb37e4c963177732ba9640fd0c81396e8aa0c9a6463ce10c00cbb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33357, "end_char_idx": 35618, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In PyTorch, the
name of a layer is defined by the name of the class attribute you give the layer. Let's define a dummy model in
PyTorch, called `SimpleModel` as follows:

```python
from torch import nn


class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.dense = nn.Linear(10, 10)
        self.intermediate = nn.Linear(10, 10)
        self.layer_norm = nn.LayerNorm(10)
```

Now we can create an instance of this model definition which will fill all weights: `dense`, `intermediate`,
`layer_norm` with random weights. We can print the model to see its architecture

```python
model = SimpleModel()

print(model)
```

This will print out the following:

```
SimpleModel(
  (dense): Linear(in_features=10, out_features=10, bias=True)
  (intermediate): Linear(in_features=10, out_features=10, bias=True)
  (layer_norm): LayerNorm((10,), eps=1e-05, elementwise_affine=True)
)
```

We can see that the layer names are defined by the name of the class attribute in PyTorch. You can print out the weight
values of a specific layer:

```python
print(model.dense.weight.data)
```

to see that the weights were randomly initialized

```
tensor([[-0.0818,  0.2207, -0.0749, -0.0030,  0.0045, -0.1569, -0.1598,  0.0212,
         -0.2077,  0.2157],
        [ 0.1044,  0.0201,  0.0990,  0.2482,  0.3116,  0.2509,  0.2866, -0.2190,
          0.2166, -0.0212],
        [-0.2000,  0.1107, -0.1999, -0.3119,  0.1559,  0.0993,  0.1776, -0.1950,
         -0.1023, -0.0447],
        [-0.0888, -0.1092,  0.2281,  0.0336,  0.1817, -0.0115,  0.2096,  0.1415,
         -0.1876, -0.2467],
        [ 0.2208, -0.2352, -0.1426, -0.2636, -0.2889, -0.2061, -0.2849, -0.0465,
          0.2577,  0.0402],
        [ 0.1502,  0.2465,  0.2566,  0.0693,  0.2352, -0.0530,  0.1859, -0.0604,
          0.2132,  0.1680],
        [ 0.1733, -0.2407, -0.1721,  0.1484,  0.0358, -0.0633, -0.0721, -0.0090,
          0.2707, -0.2509],
        [-0.1173,  0.1561,  0.2945,  0.0595, -0.1996,  0.2988, -0.0802,  0.0407,
          0.1829, -0.1568],
        [-0.1164, -0.2228, -0.0403,  0.0428,  0.1339,  0.0047,  0.1967,  0.2923,
          0.0333, -0.0536],
        [-0.1492, -0.1616,  0.1057,  0.1950, -0.2807, -0.2710, -0.1586,  0.0739,
          0.2220,  0.2358]]).


key: _node_content
string_value:
{"id_": "a006fa83-d534-444b-b48f-b0c1c04981bc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2e91d56-6c47-4f49-817d-ff81a4f44c12", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "1653a302ce09457180eb3ee3d403a09912e2c6447e20111723d5a1e8d3c00884", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b56e6a7e-51c2-4423-be46-da8b4c6bb97b", "node_type": "1", "metadata": {}, "hash": "1eb8bdbc68f31d016c44971931cb0e0b77d67b3197d8f4d9164c75635f50ff1e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35619, "end_char_idx": 38901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

In the conversion script, you should fill those randomly initialized weights with the exact weights of the
corresponding layer in the checkpoint. *E.g.*

```python
# retrieve matching layer weights, e.g. by
# recursive algorithm
layer_name = "dense"
pretrained_weight = array_of_dense_layer

model_pointer = getattr(model, "dense")

model_pointer.weight.data = torch.from_numpy(pretrained_weight)
```

While doing so, you must verify that each randomly initialized weight of your PyTorch model and its corresponding
pretrained checkpoint weight exactly match in both **shape and name**. To do so, it is **necessary** to add assert
statements for the shape and print out the names of the checkpoints weights. E.g. you should add statements like:

```python
assert (
    model_pointer.weight.shape == pretrained_weight.shape
), f"Pointer shape of random weight {model_pointer.shape} and array shape of checkpoint weight {pretrained_weight.shape} mismatched"
```

Besides, you should also print out the names of both weights to make sure they match, *e.g.*

```python
logger.info(f"Initialize PyTorch weight {layer_name} from {pretrained_weight.name}")
```

If either the shape or the name doesn't match, you probably assigned the wrong checkpoint weight to a randomly
initialized layer of the 🤗 Transformers implementation.

An incorrect shape is most likely due to an incorrect setting of the config parameters in `BrandNewBertConfig()` that
do not exactly match those that were used for the checkpoint you want to convert. However, it could also be that
PyTorch's implementation of a layer requires the weight to be transposed beforehand.

Finally, you should also check that **all** required weights are initialized and print out all checkpoint weights that
were not used for initialization to make sure the model is correctly converted. It is completely normal, that the
conversion trials fail with either a wrong shape statement or a wrong name assignment. This is most likely because either
you used incorrect parameters in `BrandNewBertConfig()`, have a wrong architecture in the 🤗 Transformers
implementation, you have a bug in the `init()` functions of one of the components of the 🤗 Transformers
implementation or you need to transpose one of the checkpoint weights.

This step should be iterated with the previous step until all weights of the checkpoint are correctly loaded in the
Transformers model. Having correctly loaded the checkpoint into the 🤗 Transformers implementation, you can then save
the model under a folder of your choice `/path/to/converted/checkpoint/folder` that should then contain both a
`pytorch_model.bin` file and a `config.json` file:

```python
model.save_pretrained("/path/to/converted/checkpoint/folder")
```

**7. Implement the forward pass**

Having managed to correctly load the pretrained weights into the 🤗 Transformers implementation, you should now make
sure that the forward pass is correctly implemented. In [Get familiar with the original repository](#3-4-run-a-pretrained-checkpoint-using-the-original-repository), you have already created a script that runs a forward
pass of the model using the original repository. Now you should write an analogous script using the 🤗 Transformers
implementation instead of the original one.


key: _node_content
string_value:
{"id_": "b56e6a7e-51c2-4423-be46-da8b4c6bb97b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a006fa83-d534-444b-b48f-b0c1c04981bc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "b2c0164edb9c4a63a308faab12e4616eebd84072d12d80a9b83c10f00983a1c0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8b425625-8e1e-4b7f-a855-c35ee9e4500d", "node_type": "1", "metadata": {}, "hash": "5334815d10b82c623f1d6822a7028d94bfda04723221d7bd352af05b898a215a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37147, "end_char_idx": 40667, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, it could also be that
PyTorch's implementation of a layer requires the weight to be transposed beforehand.

Finally, you should also check that **all** required weights are initialized and print out all checkpoint weights that
were not used for initialization to make sure the model is correctly converted. It is completely normal, that the
conversion trials fail with either a wrong shape statement or a wrong name assignment. This is most likely because either
you used incorrect parameters in `BrandNewBertConfig()`, have a wrong architecture in the 🤗 Transformers
implementation, you have a bug in the `init()` functions of one of the components of the 🤗 Transformers
implementation or you need to transpose one of the checkpoint weights.

This step should be iterated with the previous step until all weights of the checkpoint are correctly loaded in the
Transformers model. Having correctly loaded the checkpoint into the 🤗 Transformers implementation, you can then save
the model under a folder of your choice `/path/to/converted/checkpoint/folder` that should then contain both a
`pytorch_model.bin` file and a `config.json` file:

```python
model.save_pretrained("/path/to/converted/checkpoint/folder")
```

**7. Implement the forward pass**

Having managed to correctly load the pretrained weights into the 🤗 Transformers implementation, you should now make
sure that the forward pass is correctly implemented. In [Get familiar with the original repository](#3-4-run-a-pretrained-checkpoint-using-the-original-repository), you have already created a script that runs a forward
pass of the model using the original repository. Now you should write an analogous script using the 🤗 Transformers
implementation instead of the original one. It should look as follows:

```python
model = BrandNewBertModel.from_pretrained("/path/to/converted/checkpoint/folder")
input_ids = [0, 4, 4, 3, 2, 4, 1, 7, 19]
output = model(input_ids).last_hidden_states
```

It is very likely that the 🤗 Transformers implementation and the original model implementation don't give the exact
same output the very first time or that the forward pass throws an error. Don't be disappointed - it's expected! First,
you should make sure that the forward pass doesn't throw any errors. It often happens that the wrong dimensions are
used leading to a *Dimensionality mismatch* error or that the wrong data type object is used, *e.g.* `torch.long`
instead of `torch.float32`. Don't hesitate to ask the Hugging Face team for help, if you don't manage to solve
certain errors.

The final part to make sure the 🤗 Transformers implementation works correctly is to ensure that the outputs are
equivalent to a precision of `1e-3`. First, you should ensure that the output shapes are identical, *i.e.*
`outputs.shape` should yield the same value for the script of the 🤗 Transformers implementation and the original
implementation. Next, you should make sure that the output values are identical as well. This one of the most difficult
parts of adding a new model. Common mistakes why the outputs are not identical are:

- Some layers were not added, *i.e.* an *activation* layer was not added, or the residual connection was forgotten
- The word embedding matrix was not tied
- The wrong positional embeddings are used because the original implementation uses on offset
- Dropout is applied during the forward pass. To fix this make sure *model.training is False* and that no dropout
  layer is falsely activated during the forward pass, *i.e.


key: _node_content
string_value:
{"id_": "8b425625-8e1e-4b7f-a855-c35ee9e4500d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b56e6a7e-51c2-4423-be46-da8b4c6bb97b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "af050468cd3d7414bc1b8766e57295785252abf8d44d229e8e1d025adf7e32b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "97a1a8dd-a585-49e1-8ff4-a3b998e294a7", "node_type": "1", "metadata": {}, "hash": "9b16a2c9b53d32006f57f6fadb5eb275d23da687f1e3cdb18e4db61108186782", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39303, "end_char_idx": 42701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Don't be disappointed - it's expected! First,
you should make sure that the forward pass doesn't throw any errors. It often happens that the wrong dimensions are
used leading to a *Dimensionality mismatch* error or that the wrong data type object is used, *e.g.* `torch.long`
instead of `torch.float32`. Don't hesitate to ask the Hugging Face team for help, if you don't manage to solve
certain errors.

The final part to make sure the 🤗 Transformers implementation works correctly is to ensure that the outputs are
equivalent to a precision of `1e-3`. First, you should ensure that the output shapes are identical, *i.e.*
`outputs.shape` should yield the same value for the script of the 🤗 Transformers implementation and the original
implementation. Next, you should make sure that the output values are identical as well. This one of the most difficult
parts of adding a new model. Common mistakes why the outputs are not identical are:

- Some layers were not added, *i.e.* an *activation* layer was not added, or the residual connection was forgotten
- The word embedding matrix was not tied
- The wrong positional embeddings are used because the original implementation uses on offset
- Dropout is applied during the forward pass. To fix this make sure *model.training is False* and that no dropout
  layer is falsely activated during the forward pass, *i.e.* pass *self.training* to [PyTorch's functional dropout](https://pytorch.org/docs/stable/nn.functional.html?highlight=dropout#torch.nn.functional.dropout)

The best way to fix the problem is usually to look at the forward pass of the original implementation and the 🤗
Transformers implementation side-by-side and check if there are any differences. Ideally, you should debug/print out
intermediate outputs of both implementations of the forward pass to find the exact position in the network where the 🤗
Transformers implementation shows a different output than the original implementation. First, make sure that the
hard-coded `input_ids` in both scripts are identical. Next, verify that the outputs of the first transformation of
the `input_ids` (usually the word embeddings) are identical. And then work your way up to the very last layer of the
network. At some point, you will notice a difference between the two implementations, which should point you to the bug
in the 🤗 Transformers implementation. From our experience, a simple and efficient way is to add many print statements
in both the original implementation and 🤗 Transformers implementation, at the same positions in the network
respectively, and to successively remove print statements showing the same values for intermediate presentations.

When you're confident that both implementations yield the same output, verify the outputs with
`torch.allclose(original_output, output, atol=1e-3)`, you're done with the most difficult part! Congratulations - the
work left to be done should be a cakewalk 😊.

**8. Adding all necessary model tests**

At this point, you have successfully added a new model. However, it is very much possible that the model does not yet
fully comply with the required design. To make sure, the implementation is fully compatible with 🤗 Transformers, all
common tests should pass. The Cookiecutter should have automatically added a test file for your model, probably under
the same `tests/models/brand_new_bert/test_modeling_brand_new_bert.py`.


key: _node_content
string_value:
{"id_": "97a1a8dd-a585-49e1-8ff4-a3b998e294a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8b425625-8e1e-4b7f-a855-c35ee9e4500d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "10f3f215a80510be07cafb8cbd83f12ee4cd5c97db065681ec82d60bfe04847f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6084d1ea-1b3e-4b5c-aa89-bc0ebad5aa12", "node_type": "1", "metadata": {}, "hash": "05d9ba81c31cc571cddc095781768c24359b4547ab1a90e63ac426d8383af90f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41016, "end_char_idx": 44440, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Ideally, you should debug/print out
intermediate outputs of both implementations of the forward pass to find the exact position in the network where the 🤗
Transformers implementation shows a different output than the original implementation. First, make sure that the
hard-coded `input_ids` in both scripts are identical. Next, verify that the outputs of the first transformation of
the `input_ids` (usually the word embeddings) are identical. And then work your way up to the very last layer of the
network. At some point, you will notice a difference between the two implementations, which should point you to the bug
in the 🤗 Transformers implementation. From our experience, a simple and efficient way is to add many print statements
in both the original implementation and 🤗 Transformers implementation, at the same positions in the network
respectively, and to successively remove print statements showing the same values for intermediate presentations.

When you're confident that both implementations yield the same output, verify the outputs with
`torch.allclose(original_output, output, atol=1e-3)`, you're done with the most difficult part! Congratulations - the
work left to be done should be a cakewalk 😊.

**8. Adding all necessary model tests**

At this point, you have successfully added a new model. However, it is very much possible that the model does not yet
fully comply with the required design. To make sure, the implementation is fully compatible with 🤗 Transformers, all
common tests should pass. The Cookiecutter should have automatically added a test file for your model, probably under
the same `tests/models/brand_new_bert/test_modeling_brand_new_bert.py`. Run this test file to verify that all common
tests pass:

```bash
pytest tests/models/brand_new_bert/test_modeling_brand_new_bert.py
```

Having fixed all common tests, it is now crucial to ensure that all the nice work you have done is well tested, so that

- a) The community can easily understand your work by looking at specific tests of *brand_new_bert*
- b) Future changes to your model will not break any important feature of the model.

At first, integration tests should be added. Those integration tests essentially do the same as the debugging scripts
you used earlier to implement the model to 🤗 Transformers. A template of those model tests has already added by the
Cookiecutter, called `BrandNewBertModelIntegrationTests` and only has to be filled out by you. To ensure that those
tests are passing, run

```bash
RUN_SLOW=1 pytest -sv tests/models/brand_new_bert/test_modeling_brand_new_bert.py::BrandNewBertModelIntegrationTests
```

<Tip>

In case you are using Windows, you should replace `RUN_SLOW=1` with `SET RUN_SLOW=1`

</Tip>

Second, all features that are special to *brand_new_bert* should be tested additionally in a separate test under
`BrandNewBertModelTester`/`BrandNewBertModelTest`. This part is often forgotten but is extremely useful in two
ways:

- It helps to transfer the knowledge you have acquired during the model addition to the community by showing how the
  special features of *brand_new_bert* should work.
- Future contributors can quickly test changes to the model by running those special tests.


**9. Implement the tokenizer**

Next, we should add the tokenizer of *brand_new_bert*. Usually, the tokenizer is equivalent to or very similar to an
already existing tokenizer of 🤗 Transformers.


key: _node_content
string_value:
{"id_": "6084d1ea-1b3e-4b5c-aa89-bc0ebad5aa12", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "97a1a8dd-a585-49e1-8ff4-a3b998e294a7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "ed2d002ed644782732385207609108e33ad3d608d7a3e573bd630d14f96d3848", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "44241343-2e7d-4115-bcfb-57e3b285073e", "node_type": "1", "metadata": {}, "hash": "b1f7182610ff5e7000eb4d67ae7bf2b76a40d9415a374d494774d07bb476ccb8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 43147, "end_char_idx": 46452, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
At first, integration tests should be added. Those integration tests essentially do the same as the debugging scripts
you used earlier to implement the model to 🤗 Transformers. A template of those model tests has already added by the
Cookiecutter, called `BrandNewBertModelIntegrationTests` and only has to be filled out by you. To ensure that those
tests are passing, run

```bash
RUN_SLOW=1 pytest -sv tests/models/brand_new_bert/test_modeling_brand_new_bert.py::BrandNewBertModelIntegrationTests
```

<Tip>

In case you are using Windows, you should replace `RUN_SLOW=1` with `SET RUN_SLOW=1`

</Tip>

Second, all features that are special to *brand_new_bert* should be tested additionally in a separate test under
`BrandNewBertModelTester`/`BrandNewBertModelTest`. This part is often forgotten but is extremely useful in two
ways:

- It helps to transfer the knowledge you have acquired during the model addition to the community by showing how the
  special features of *brand_new_bert* should work.
- Future contributors can quickly test changes to the model by running those special tests.


**9. Implement the tokenizer**

Next, we should add the tokenizer of *brand_new_bert*. Usually, the tokenizer is equivalent to or very similar to an
already existing tokenizer of 🤗 Transformers.

It is very important to find/extract the original tokenizer file and to manage to load this file into the 🤗
Transformers' implementation of the tokenizer.

To ensure that the tokenizer works correctly, it is recommended to first create a script in the original repository
that inputs a string and returns the `input_ids`. It could look similar to this (in pseudo-code):

```python
input_str = "This is a long example input string containing special characters .$?-, numbers 2872 234 12 and words."
model = BrandNewBertModel.load_pretrained_checkpoint("/path/to/checkpoint/")
input_ids = model.tokenize(input_str)
```

You might have to take a deeper look again into the original repository to find the correct tokenizer function or you
might even have to do changes to your clone of the original repository to only output the `input_ids`. Having written
a functional tokenization script that uses the original repository, an analogous script for 🤗 Transformers should be
created. It should look similar to this:

```python
from transformers import BrandNewBertTokenizer

input_str = "This is a long example input string containing special characters .$?-, numbers 2872 234 12 and words."

tokenizer = BrandNewBertTokenizer.from_pretrained("/path/to/tokenizer/folder/")

input_ids = tokenizer(input_str).input_ids
```

When both `input_ids` yield the same values, as a final step a tokenizer test file should also be added.

Analogous to the modeling test files of *brand_new_bert*, the tokenization test files of *brand_new_bert* should
contain a couple of hard-coded integration tests.

**10. Run End-to-end integration tests**

Having added the tokenizer, you should also add a couple of end-to-end integration tests using both the model and the
tokenizer to `tests/models/brand_new_bert/test_modeling_brand_new_bert.py` in 🤗 Transformers.
Such a test should show on a meaningful
text-to-text sample that the 🤗 Transformers implementation works as expected. A meaningful text-to-text sample can
include *e.g.


key: _node_content
string_value:
{"id_": "44241343-2e7d-4115-bcfb-57e3b285073e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6084d1ea-1b3e-4b5c-aa89-bc0ebad5aa12", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "8537ed84ac95a2b5f9263ec9ecf6e4a9d3a3e2c51c631386db93bd79a1589911", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "22d96ae7-3f78-4760-a40d-fb7c2e7b4e39", "node_type": "1", "metadata": {}, "hash": "b578a32d139f003213b9aeadf7461667bf2e8219bd4a3dba8834d879e11cf3f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 44764, "end_char_idx": 48147, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It could look similar to this (in pseudo-code):

```python
input_str = "This is a long example input string containing special characters .$?-, numbers 2872 234 12 and words."
model = BrandNewBertModel.load_pretrained_checkpoint("/path/to/checkpoint/")
input_ids = model.tokenize(input_str)
```

You might have to take a deeper look again into the original repository to find the correct tokenizer function or you
might even have to do changes to your clone of the original repository to only output the `input_ids`. Having written
a functional tokenization script that uses the original repository, an analogous script for 🤗 Transformers should be
created. It should look similar to this:

```python
from transformers import BrandNewBertTokenizer

input_str = "This is a long example input string containing special characters .$?-, numbers 2872 234 12 and words."

tokenizer = BrandNewBertTokenizer.from_pretrained("/path/to/tokenizer/folder/")

input_ids = tokenizer(input_str).input_ids
```

When both `input_ids` yield the same values, as a final step a tokenizer test file should also be added.

Analogous to the modeling test files of *brand_new_bert*, the tokenization test files of *brand_new_bert* should
contain a couple of hard-coded integration tests.

**10. Run End-to-end integration tests**

Having added the tokenizer, you should also add a couple of end-to-end integration tests using both the model and the
tokenizer to `tests/models/brand_new_bert/test_modeling_brand_new_bert.py` in 🤗 Transformers.
Such a test should show on a meaningful
text-to-text sample that the 🤗 Transformers implementation works as expected. A meaningful text-to-text sample can
include *e.g.* a source-to-target-translation pair, an article-to-summary pair, a question-to-answer pair, etc… If none
of the ported checkpoints has been fine-tuned on a downstream task it is enough to simply rely on the model tests. In a
final step to ensure that the model is fully functional, it is advised that you also run all tests on GPU. It can
happen that you forgot to add some `.to(self.device)` statements to internal tensors of the model, which in such a
test would show in an error. In case you have no access to a GPU, the Hugging Face team can take care of running those
tests for you.

**11. Add Docstring**

Now, all the necessary functionality for *brand_new_bert* is added - you're almost done! The only thing left to add is
a nice docstring and a doc page. The Cookiecutter should have added a template file called
`docs/source/model_doc/brand_new_bert.md` that you should fill out. Users of your model will usually first look at
this page before using your model. Hence, the documentation must be understandable and concise. It is very useful for
the community to add some *Tips* to show how the model should be used. Don't hesitate to ping the Hugging Face team
regarding the docstrings.

Next, make sure that the docstring added to `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` is
correct and included all necessary inputs and outputs. We have a detailed guide about writing documentation and our docstring format [here](writing-documentation). It is always good to remind oneself that documentation should
be treated at least as carefully as the code in 🤗 Transformers since the documentation is usually the first contact
point of the community with the model.


key: _node_content
string_value:
{"id_": "22d96ae7-3f78-4760-a40d-fb7c2e7b4e39", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "44241343-2e7d-4115-bcfb-57e3b285073e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "3a8ddf295a129d62aca02068c3e3131dc9fc02afa088b413dac735ba5be75c30", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c2ecd359-3026-4961-b66b-491a74156820", "node_type": "1", "metadata": {}, "hash": "9f82f4b7d877fc9637c4cfef639a74321f1dffdea649a7151c866df39e43e489", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 46452, "end_char_idx": 49833, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* a source-to-target-translation pair, an article-to-summary pair, a question-to-answer pair, etc… If none
of the ported checkpoints has been fine-tuned on a downstream task it is enough to simply rely on the model tests. In a
final step to ensure that the model is fully functional, it is advised that you also run all tests on GPU. It can
happen that you forgot to add some `.to(self.device)` statements to internal tensors of the model, which in such a
test would show in an error. In case you have no access to a GPU, the Hugging Face team can take care of running those
tests for you.

**11. Add Docstring**

Now, all the necessary functionality for *brand_new_bert* is added - you're almost done! The only thing left to add is
a nice docstring and a doc page. The Cookiecutter should have added a template file called
`docs/source/model_doc/brand_new_bert.md` that you should fill out. Users of your model will usually first look at
this page before using your model. Hence, the documentation must be understandable and concise. It is very useful for
the community to add some *Tips* to show how the model should be used. Don't hesitate to ping the Hugging Face team
regarding the docstrings.

Next, make sure that the docstring added to `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` is
correct and included all necessary inputs and outputs. We have a detailed guide about writing documentation and our docstring format [here](writing-documentation). It is always good to remind oneself that documentation should
be treated at least as carefully as the code in 🤗 Transformers since the documentation is usually the first contact
point of the community with the model.

**Code refactor**

Great, now you have added all the necessary code for *brand_new_bert*. At this point, you should correct some potential
incorrect code style by running:

```bash
make style
```

and verify that your coding style passes the quality check:

```bash
make quality
```

There are a couple of other very strict design tests in 🤗 Transformers that might still be failing, which shows up in
the tests of your pull request. This is often because of some missing information in the docstring or some incorrect
naming. The Hugging Face team will surely help you if you're stuck here.

Lastly, it is always a good idea to refactor one's code after having ensured that the code works correctly. With all
tests passing, now it's a good time to go over the added code again and do some refactoring.

You have now finished the coding part, congratulation! 🎉 You are Awesome! 😎

**12. Upload the models to the model hub**

In this final part, you should convert and upload all checkpoints to the model hub and add a model card for each
uploaded model checkpoint. You can get familiar with the hub functionalities by reading our [Model sharing and uploading Page](model_sharing). You should work alongside the Hugging Face team here to decide on a fitting name for each
checkpoint and to get the required access rights to be able to upload the model under the author's organization of
*brand_new_bert*. The `push_to_hub` method, present in all models in `transformers`, is a quick and efficient way to push your checkpoint to the hub. A little snippet is pasted below:

```python
brand_new_bert.push_to_hub("brand_new_bert")
# Uncomment the following line to push to an organization.


key: _node_content
string_value:
{"id_": "c2ecd359-3026-4961-b66b-491a74156820", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5db7d266-916a-4160-887d-ef65997b0cae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "22d96ae7-3f78-4760-a40d-fb7c2e7b4e39", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers"}, "hash": "779c0ee32fc50d0bc1f2450b15ded0fe0f4e28e70ce0dd5bc5dcb7e2d156ccbc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 48149, "end_char_idx": 51422, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Code refactor**

Great, now you have added all the necessary code for *brand_new_bert*. At this point, you should correct some potential
incorrect code style by running:

```bash
make style
```

and verify that your coding style passes the quality check:

```bash
make quality
```

There are a couple of other very strict design tests in 🤗 Transformers that might still be failing, which shows up in
the tests of your pull request. This is often because of some missing information in the docstring or some incorrect
naming. The Hugging Face team will surely help you if you're stuck here.

Lastly, it is always a good idea to refactor one's code after having ensured that the code works correctly. With all
tests passing, now it's a good time to go over the added code again and do some refactoring.

You have now finished the coding part, congratulation! 🎉 You are Awesome! 😎

**12. Upload the models to the model hub**

In this final part, you should convert and upload all checkpoints to the model hub and add a model card for each
uploaded model checkpoint. You can get familiar with the hub functionalities by reading our [Model sharing and uploading Page](model_sharing). You should work alongside the Hugging Face team here to decide on a fitting name for each
checkpoint and to get the required access rights to be able to upload the model under the author's organization of
*brand_new_bert*. The `push_to_hub` method, present in all models in `transformers`, is a quick and efficient way to push your checkpoint to the hub. A little snippet is pasted below:

```python
brand_new_bert.push_to_hub("brand_new_bert")
# Uncomment the following line to push to an organization.
# brand_new_bert.push_to_hub("<organization>/brand_new_bert")
```

It is worth spending some time to create fitting model cards for each checkpoint. The model cards should highlight the
specific characteristics of this particular checkpoint, *e.g.* On which dataset was the checkpoint
pretrained/fine-tuned on? On what down-stream task should the model be used? And also include some code on how to
correctly use the model.

**13. (Optional) Add notebook**

It is very helpful to add a notebook that showcases in-detail how *brand_new_bert* can be used for inference and/or
fine-tuned on a downstream task. This is not mandatory to merge your PR, but very useful for the community.

**14. Submit your finished PR**

You're done programming now and can move to the last step, which is getting your PR merged into main. Usually, the
Hugging Face team should have helped you already at this point, but it is worth taking some time to give your finished
PR a nice description and eventually add comments to your code, if you want to point out certain design choices to your
reviewer.

### Share your work!!

Now, it's time to get some credit from the community for your work! Having completed a model addition is a major
contribution to Transformers and the whole NLP community. Your code and the ported pre-trained models will certainly be
used by hundreds and possibly even thousands of developers and researchers. You should be proud of your work and share
your achievements with the community.

**You have made another model that is super easy to access for everyone in the community! 🤯**


key: _node_content
string_value:
{"id_": "e3c49845-cb14-4a13-bb3e-9aeef8dd5ed5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "014f695b-a4d3-4474-94ce-19680b0f6063", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers"}, "hash": "4833f52239d8ecf22c2fa5e74690dbc8e6cee4387648bcf70bbe622fd6edc308", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4bb241e3-d6bb-4688-9488-967f97e17c24", "node_type": "1", "metadata": {}, "hash": "31874c9931bb2a115e2638d7c5b9a26e10bc5f1f8f2cd2db3824b9ae774bdac2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2361, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!--⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

# Using pipelines for a webserver

<Tip>
Creating an inference engine is a complex topic, and the "best" solution 
will most likely depend on your problem space. Are you on CPU or GPU? Do
you want the lowest latency, the highest throughput, support for
many models, or just highly optimize 1 specific model?
There are many ways to tackle this topic, so what we are going to present is a good default
to get started which may not necessarily be the most optimal solution for you.
</Tip>


The key thing to understand is that we can use an iterator, just like you would [on a
dataset](pipeline_tutorial#using-pipelines-on-a-dataset), since a webserver is basically a system that waits for requests and
treats them as they come in.

Usually webservers are multiplexed (multithreaded, async, etc..) to handle various
requests concurrently. Pipelines on the other hand (and mostly the underlying models)
are not really great for parallelism; they take up a lot of RAM, so it's best to give them all the available resources when they are running or it's a compute-intensive job.

We are going to solve that by having the webserver handle the light load of receiving
and sending requests, and having a single thread handling the actual work.
This example is going to use `starlette`. The actual framework is not really
important, but you might have to tune or change the code if you are using another
one to achieve the same effect.

Create `server.py`:

```py
from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route
from transformers import pipeline
import asyncio


async def homepage(request):
    payload = await request.body()
    string = payload.decode("utf-8")
    response_q = asyncio.Queue()
    await request.app.model_queue.put((string, response_q))
    output = await response_q.get()
    return JSONResponse(output)


async def server_loop(q):
    pipe = pipeline(model="google-bert/bert-base-uncased")
    while True:
        (string, response_q) = await q.get()
        out = pipe(string)
        await response_q.put(out)


app = Starlette(
    routes=[
        Route("/", homepage, methods=["POST"]),
    ],
)


key: _node_content
string_value:
{"id_": "4bb241e3-d6bb-4688-9488-967f97e17c24", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "014f695b-a4d3-4474-94ce-19680b0f6063", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers"}, "hash": "4833f52239d8ecf22c2fa5e74690dbc8e6cee4387648bcf70bbe622fd6edc308", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3c49845-cb14-4a13-bb3e-9aeef8dd5ed5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers"}, "hash": "7ed842c217e658042d2f6531b9e7e292bc7198a3ca29657bd8f6b80c84e42389", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8487a74-e968-4b14-9b84-ea9bef3e4295", "node_type": "1", "metadata": {}, "hash": "86fe1179852aefe265940043d5a2d81a7ff6bf32971e1850fd31cc2fdaea0569", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 659, "end_char_idx": 4351, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The key thing to understand is that we can use an iterator, just like you would [on a
dataset](pipeline_tutorial#using-pipelines-on-a-dataset), since a webserver is basically a system that waits for requests and
treats them as they come in.

Usually webservers are multiplexed (multithreaded, async, etc..) to handle various
requests concurrently. Pipelines on the other hand (and mostly the underlying models)
are not really great for parallelism; they take up a lot of RAM, so it's best to give them all the available resources when they are running or it's a compute-intensive job.

We are going to solve that by having the webserver handle the light load of receiving
and sending requests, and having a single thread handling the actual work.
This example is going to use `starlette`. The actual framework is not really
important, but you might have to tune or change the code if you are using another
one to achieve the same effect.

Create `server.py`:

```py
from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route
from transformers import pipeline
import asyncio


async def homepage(request):
    payload = await request.body()
    string = payload.decode("utf-8")
    response_q = asyncio.Queue()
    await request.app.model_queue.put((string, response_q))
    output = await response_q.get()
    return JSONResponse(output)


async def server_loop(q):
    pipe = pipeline(model="google-bert/bert-base-uncased")
    while True:
        (string, response_q) = await q.get()
        out = pipe(string)
        await response_q.put(out)


app = Starlette(
    routes=[
        Route("/", homepage, methods=["POST"]),
    ],
)


@app.on_event("startup")
async def startup_event():
    q = asyncio.Queue()
    app.model_queue = q
    asyncio.create_task(server_loop(q))
```

Now you can start it with:
```bash
uvicorn server:app
```

And you can query it:
```bash
curl -X POST -d "test [MASK]" http://localhost:8000/
#[{"score":0.7742936015129089,"token":1012,"token_str":".","sequence":"test."},...]
```

And there you go, now you have a good idea of how to create a webserver!

What is really important is that we load the model only **once**, so there are no copies
of the model on the webserver. This way, no unnecessary RAM is being used.
Then the queuing mechanism allows you to do fancy stuff like maybe accumulating a few
items before inferring to use dynamic batching:

<Tip warning={true}>

The code sample below is intentionally written like pseudo-code for readability.
Do not run this without checking if it makes sense for your system resources!

</Tip>

```py
(string, rq) = await q.get()
strings = []
queues = []
while True:
    try:
        (string, rq) = await asyncio.wait_for(q.get(), timeout=0.001)  # 1ms
    except asyncio.exceptions.TimeoutError:
        break
    strings.append(string)
    queues.append(rq)
strings
outs = pipe(strings, batch_size=len(strings))
for rq, out in zip(queues, outs):
    await rq.put(out)
```

Again, the proposed code is optimized for readability, not for being the best code.
First of all, there's no batch size limit which is usually not a 
great idea. Next, the timeout is reset on every queue fetch, meaning you could
wait much more than 1ms before running the inference (delaying the first request 
by that much). 

It would be better to have a single 1ms deadline.

This will always wait for 1ms even if the queue is empty, which might not be the
best since you probably want to start doing inference if there's nothing in the queue.
But maybe it does make sense if batching is really crucial for your use case.
Again, there's really no one best solution.


key: _node_content
string_value:
{"id_": "f8487a74-e968-4b14-9b84-ea9bef3e4295", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "014f695b-a4d3-4474-94ce-19680b0f6063", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers"}, "hash": "4833f52239d8ecf22c2fa5e74690dbc8e6cee4387648bcf70bbe622fd6edc308", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4bb241e3-d6bb-4688-9488-967f97e17c24", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers"}, "hash": "ccbe9e805dc59f5a009ae137bb6daed2dd5679278091b5a7e55afad0766cc661", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4354, "end_char_idx": 6456, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Few things you might want to consider

### Error checking

There's a lot that can go wrong in production: out of memory, out of space,
loading the model might fail, the query might be wrong, the query might be
correct but still fail to run because of a model misconfiguration, and so on.

Generally, it's good if the server outputs the errors to the user, so
adding a lot of `try..except` statements to show those errors is a good
idea. But keep in mind it may also be a security risk to reveal all those errors depending 
on your security context.

### Circuit breaking

Webservers usually look better when they do circuit breaking. It means they 
return proper errors when they're overloaded instead of just waiting for the query indefinitely. Return a 503 error instead of waiting for a super long time or a 504 after a long time.

This is relatively easy to implement in the proposed code since there is a single queue.
Looking at the queue size is a basic way to start returning errors before your 
webserver fails under load.

### Blocking the main thread

Currently PyTorch is not async aware, and computation will block the main
thread while running. That means it would be better if PyTorch was forced to run
on its own thread/process. This wasn't done here because the code is a lot more
complex (mostly because threads and async and queues don't play nice together).
But ultimately it does the same thing.

This would be important if the inference of single items were long (> 1s) because 
in this case, it means every query during inference would have to wait for 1s before
even receiving an error.

### Dynamic batching

In general, batching is not necessarily an improvement over passing 1 item at 
a time (see [batching details](./main_classes/pipelines#pipeline-batching) for more information). But it can be very effective
when used in the correct setting. In the API, there is no dynamic
batching by default (too much opportunity for a slowdown). But for BLOOM inference -
which is a very large model - dynamic batching is **essential** to provide a decent experience for everyone.


key: _node_content
string_value:
{"id_": "84ee3cde-95ad-4b84-b3c7-95174a2787ac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25001e7e-45df-4a9a-82fe-97aa2b82ea0d", "node_type": "1", "metadata": {}, "hash": "db3529fe6abeda9c5d241bad9e2a829cc3ea91c3fdedb579edc6c756ecd10fb9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2987, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Summary of the tokenizers

[[open-in-colab]]

On this page, we will have a closer look at tokenization.

<Youtube id="VFp38yj8h3A"/>

As we saw in [the preprocessing tutorial](preprocessing), tokenizing a text is splitting it into words or
subwords, which then are converted to ids through a look-up table. Converting words or subwords to ids is
straightforward, so in this summary, we will focus on splitting a text into words or subwords (i.e. tokenizing a text).
More specifically, we will look at the three main types of tokenizers used in 🤗 Transformers: [Byte-Pair Encoding
(BPE)](#byte-pair-encoding), [WordPiece](#wordpiece), and [SentencePiece](#sentencepiece), and show examples
of which tokenizer type is used by which model.

Note that on each model page, you can look at the documentation of the associated tokenizer to know which tokenizer
type was used by the pretrained model. For instance, if we look at [`BertTokenizer`], we can see
that the model uses [WordPiece](#wordpiece).

## Introduction

Splitting a text into smaller chunks is a task that is harder than it looks, and there are multiple ways of doing so.
For instance, let's look at the sentence `"Don't you love 🤗 Transformers? We sure do."`

<Youtube id="nhJxYji1aho"/>

A simple way of tokenizing this text is to split it by spaces, which would give:

```
["Don't", "you", "love", "🤗", "Transformers?", "We", "sure", "do."]
```

This is a sensible first step, but if we look at the tokens `"Transformers?"` and `"do."`, we notice that the
punctuation is attached to the words `"Transformer"` and `"do"`, which is suboptimal. We should take the
punctuation into account so that a model does not have to learn a different representation of a word and every possible
punctuation symbol that could follow it, which would explode the number of representations the model has to learn.
Taking punctuation into account, tokenizing our exemplary text would give:

```
["Don", "'", "t", "you", "love", "🤗", "Transformers", "?", "We", "sure", "do", "."]
```

Better. However, it is disadvantageous, how the tokenization dealt with the word `"Don't"`. `"Don't"` stands for
`"do not"`, so it would be better tokenized as `["Do", "n't"]`. This is where things start getting complicated, and
part of the reason each model has its own tokenizer type. Depending on the rules we apply for tokenizing a text, a
different tokenized output is generated for the same text. A pretrained model only performs properly if you feed it an
input that was tokenized with the same rules that were used to tokenize its training data.

[spaCy](https://spacy.io/) and [Moses](http://www.statmt.org/moses/?n=Development.GetStarted) are two popular
rule-based tokenizers. Applying them on our example, *spaCy* and *Moses* would output something like:

```
["Do", "n't", "you", "love", "🤗", "Transformers", "?", "We", "sure", "do", "."]
```

As can be seen space and punctuation tokenization, as well as rule-based tokenization, is used here.


key: _node_content
string_value:
{"id_": "25001e7e-45df-4a9a-82fe-97aa2b82ea0d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "84ee3cde-95ad-4b84-b3c7-95174a2787ac", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "b4107365c574a29fd3ecf651cd1c9ea038b3132b0411c805270aef3c8a552123", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a6964189-721b-4bb9-8f07-6efbad181a44", "node_type": "1", "metadata": {}, "hash": "12952e2eeb2e588dcf770a55cc4742f6cb495c4da0b3005a36eed67e9427fe04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1486, "end_char_idx": 4839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"` and `"do."`, we notice that the
punctuation is attached to the words `"Transformer"` and `"do"`, which is suboptimal. We should take the
punctuation into account so that a model does not have to learn a different representation of a word and every possible
punctuation symbol that could follow it, which would explode the number of representations the model has to learn.
Taking punctuation into account, tokenizing our exemplary text would give:

```
["Don", "'", "t", "you", "love", "🤗", "Transformers", "?", "We", "sure", "do", "."]
```

Better. However, it is disadvantageous, how the tokenization dealt with the word `"Don't"`. `"Don't"` stands for
`"do not"`, so it would be better tokenized as `["Do", "n't"]`. This is where things start getting complicated, and
part of the reason each model has its own tokenizer type. Depending on the rules we apply for tokenizing a text, a
different tokenized output is generated for the same text. A pretrained model only performs properly if you feed it an
input that was tokenized with the same rules that were used to tokenize its training data.

[spaCy](https://spacy.io/) and [Moses](http://www.statmt.org/moses/?n=Development.GetStarted) are two popular
rule-based tokenizers. Applying them on our example, *spaCy* and *Moses* would output something like:

```
["Do", "n't", "you", "love", "🤗", "Transformers", "?", "We", "sure", "do", "."]
```

As can be seen space and punctuation tokenization, as well as rule-based tokenization, is used here. Space and
punctuation tokenization and rule-based tokenization are both examples of word tokenization, which is loosely defined
as splitting sentences into words. While it's the most intuitive way to split texts into smaller chunks, this
tokenization method can lead to problems for massive text corpora. In this case, space and punctuation tokenization
usually generates a very big vocabulary (the set of all unique words and tokens used). *E.g.*, [Transformer XL](model_doc/transfo-xl) uses space and punctuation tokenization, resulting in a vocabulary size of 267,735!

Such a big vocabulary size forces the model to have an enormous embedding matrix as the input and output layer, which
causes both an increased memory and time complexity. In general, transformers models rarely have a vocabulary size
greater than 50,000, especially if they are pretrained only on a single language.

So if simple space and punctuation tokenization is unsatisfactory, why not simply tokenize on characters?

<Youtube id="ssLq_EK2jLE"/>

While character tokenization is very simple and would greatly reduce memory and time complexity it makes it much harder
for the model to learn meaningful input representations. *E.g.* learning a meaningful context-independent
representation for the letter `"t"` is much harder than learning a context-independent representation for the word
`"today"`. Therefore, character tokenization is often accompanied by a loss of performance. So to get the best of
both worlds, transformers models use a hybrid between word-level and character-level tokenization called **subword**
tokenization.

## Subword tokenization

<Youtube id="zHvTiHr506c"/>

Subword tokenization algorithms rely on the principle that frequently used words should not be split into smaller
subwords, but rare words should be decomposed into meaningful subwords.


key: _node_content
string_value:
{"id_": "a6964189-721b-4bb9-8f07-6efbad181a44", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "25001e7e-45df-4a9a-82fe-97aa2b82ea0d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "4d8384c600f509ba2d625704447b36a31adb723c5a02af4cc0d5c4f3f221dca6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c53c19f-e780-4e32-a121-422b1320158c", "node_type": "1", "metadata": {}, "hash": "6cd62ebc4905a3ea204968a163181fe8fd52ef3f38fb71d2404c14e9a275a605", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2988, "end_char_idx": 6346, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Space and
punctuation tokenization and rule-based tokenization are both examples of word tokenization, which is loosely defined
as splitting sentences into words. While it's the most intuitive way to split texts into smaller chunks, this
tokenization method can lead to problems for massive text corpora. In this case, space and punctuation tokenization
usually generates a very big vocabulary (the set of all unique words and tokens used). *E.g.*, [Transformer XL](model_doc/transfo-xl) uses space and punctuation tokenization, resulting in a vocabulary size of 267,735!

Such a big vocabulary size forces the model to have an enormous embedding matrix as the input and output layer, which
causes both an increased memory and time complexity. In general, transformers models rarely have a vocabulary size
greater than 50,000, especially if they are pretrained only on a single language.

So if simple space and punctuation tokenization is unsatisfactory, why not simply tokenize on characters?

<Youtube id="ssLq_EK2jLE"/>

While character tokenization is very simple and would greatly reduce memory and time complexity it makes it much harder
for the model to learn meaningful input representations. *E.g.* learning a meaningful context-independent
representation for the letter `"t"` is much harder than learning a context-independent representation for the word
`"today"`. Therefore, character tokenization is often accompanied by a loss of performance. So to get the best of
both worlds, transformers models use a hybrid between word-level and character-level tokenization called **subword**
tokenization.

## Subword tokenization

<Youtube id="zHvTiHr506c"/>

Subword tokenization algorithms rely on the principle that frequently used words should not be split into smaller
subwords, but rare words should be decomposed into meaningful subwords. For instance `"annoyingly"` might be
considered a rare word and could be decomposed into `"annoying"` and `"ly"`. Both `"annoying"` and `"ly"` as
stand-alone subwords would appear more frequently while at the same time the meaning of `"annoyingly"` is kept by the
composite meaning of `"annoying"` and `"ly"`. This is especially useful in agglutinative languages such as Turkish,
where you can form (almost) arbitrarily long complex words by stringing together subwords.

Subword tokenization allows the model to have a reasonable vocabulary size while being able to learn meaningful
context-independent representations. In addition, subword tokenization enables the model to process words it has never
seen before, by decomposing them into known subwords. For instance, the [`~transformers.BertTokenizer`] tokenizes
`"I have a new GPU!"` as follows:

```py
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")
>>> tokenizer.tokenize("I have a new GPU!")
["i", "have", "a", "new", "gp", "##u", "!"]
```

Because we are considering the uncased model, the sentence was lowercased first. We can see that the words `["i", "have", "a", "new"]` are present in the tokenizer's vocabulary, but the word `"gpu"` is not. Consequently, the
tokenizer splits `"gpu"` into known subwords: `["gp" and "##u"]`. `"##"` means that the rest of the token should
be attached to the previous one, without space (for decoding or reversal of the tokenization).


key: _node_content
string_value:
{"id_": "0c53c19f-e780-4e32-a121-422b1320158c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a6964189-721b-4bb9-8f07-6efbad181a44", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "e45a10d24608efaec5d2c9638fceaa8eba9cf9f815d22dc42c3d699af61707b2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "752258fe-7bbc-4d58-b989-b97b43430db3", "node_type": "1", "metadata": {}, "hash": "c6c5e8c3eea22dd0b14b73f8fa92e985fd47836293203981f8d77ceb9d6a0938", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4840, "end_char_idx": 7687, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For instance `"annoyingly"` might be
considered a rare word and could be decomposed into `"annoying"` and `"ly"`. Both `"annoying"` and `"ly"` as
stand-alone subwords would appear more frequently while at the same time the meaning of `"annoyingly"` is kept by the
composite meaning of `"annoying"` and `"ly"`. This is especially useful in agglutinative languages such as Turkish,
where you can form (almost) arbitrarily long complex words by stringing together subwords.

Subword tokenization allows the model to have a reasonable vocabulary size while being able to learn meaningful
context-independent representations. In addition, subword tokenization enables the model to process words it has never
seen before, by decomposing them into known subwords. For instance, the [`~transformers.BertTokenizer`] tokenizes
`"I have a new GPU!"` as follows:

```py
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")
>>> tokenizer.tokenize("I have a new GPU!")
["i", "have", "a", "new", "gp", "##u", "!"]
```

Because we are considering the uncased model, the sentence was lowercased first. We can see that the words `["i", "have", "a", "new"]` are present in the tokenizer's vocabulary, but the word `"gpu"` is not. Consequently, the
tokenizer splits `"gpu"` into known subwords: `["gp" and "##u"]`. `"##"` means that the rest of the token should
be attached to the previous one, without space (for decoding or reversal of the tokenization).

As another example, [`~transformers.XLNetTokenizer`] tokenizes our previously exemplary text as follows:

```py
>>> from transformers import XLNetTokenizer

>>> tokenizer = XLNetTokenizer.from_pretrained("xlnet/xlnet-base-cased")
>>> tokenizer.tokenize("Don't you love 🤗 Transformers? We sure do.")
["▁Don", "'", "t", "▁you", "▁love", "▁", "🤗", "▁", "Transform", "ers", "?", "▁We", "▁sure", "▁do", "."]
```

We'll get back to the meaning of those `"▁"` when we look at [SentencePiece](#sentencepiece). As one can see,
the rare word `"Transformers"` has been split into the more frequent subwords `"Transform"` and `"ers"`.

Let's now look at how the different subword tokenization algorithms work. Note that all of those tokenization
algorithms rely on some form of training which is usually done on the corpus the corresponding model will be trained
on.

<a id='byte-pair-encoding'></a>

### Byte-Pair Encoding (BPE)

Byte-Pair Encoding (BPE) was introduced in [Neural Machine Translation of Rare Words with Subword Units (Sennrich et
al., 2015)](https://arxiv.org/abs/1508.07909). BPE relies on a pre-tokenizer that splits the training data into
words. Pretokenization can be as simple as space tokenization, e.g. [GPT-2](model_doc/gpt2), [RoBERTa](model_doc/roberta). More advanced pre-tokenization include rule-based tokenization, e.g.


key: _node_content
string_value:
{"id_": "752258fe-7bbc-4d58-b989-b97b43430db3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c53c19f-e780-4e32-a121-422b1320158c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "1e831cb8f1e51f6ed45baf89755398201ec0d03f7739c2c6ac3ab61da2baefc1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ffa04e0f-160d-4407-8d9f-4171185ac1e2", "node_type": "1", "metadata": {}, "hash": "b463e05c505295089811a339bbdd796d62a84bc1986c11f4ea6b017a8e413dde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6348, "end_char_idx": 8974, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As another example, [`~transformers.XLNetTokenizer`] tokenizes our previously exemplary text as follows:

```py
>>> from transformers import XLNetTokenizer

>>> tokenizer = XLNetTokenizer.from_pretrained("xlnet/xlnet-base-cased")
>>> tokenizer.tokenize("Don't you love 🤗 Transformers? We sure do.")
["▁Don", "'", "t", "▁you", "▁love", "▁", "🤗", "▁", "Transform", "ers", "?", "▁We", "▁sure", "▁do", "."]
```

We'll get back to the meaning of those `"▁"` when we look at [SentencePiece](#sentencepiece). As one can see,
the rare word `"Transformers"` has been split into the more frequent subwords `"Transform"` and `"ers"`.

Let's now look at how the different subword tokenization algorithms work. Note that all of those tokenization
algorithms rely on some form of training which is usually done on the corpus the corresponding model will be trained
on.

<a id='byte-pair-encoding'></a>

### Byte-Pair Encoding (BPE)

Byte-Pair Encoding (BPE) was introduced in [Neural Machine Translation of Rare Words with Subword Units (Sennrich et
al., 2015)](https://arxiv.org/abs/1508.07909). BPE relies on a pre-tokenizer that splits the training data into
words. Pretokenization can be as simple as space tokenization, e.g. [GPT-2](model_doc/gpt2), [RoBERTa](model_doc/roberta). More advanced pre-tokenization include rule-based tokenization, e.g. [XLM](model_doc/xlm),
[FlauBERT](model_doc/flaubert) which uses Moses for most languages, or [GPT](model_doc/openai-gpt) which uses
spaCy and ftfy, to count the frequency of each word in the training corpus.

After pre-tokenization, a set of unique words has been created and the frequency with which each word occurred in the
training data has been determined. Next, BPE creates a base vocabulary consisting of all symbols that occur in the set
of unique words and learns merge rules to form a new symbol from two symbols of the base vocabulary. It does so until
the vocabulary has attained the desired vocabulary size. Note that the desired vocabulary size is a hyperparameter to
define before training the tokenizer.

As an example, let's assume that after pre-tokenization, the following set of words including their frequency has been
determined:

```
("hug", 10), ("pug", 5), ("pun", 12), ("bun", 4), ("hugs", 5)
```

Consequently, the base vocabulary is `["b", "g", "h", "n", "p", "s", "u"]`. Splitting all words into symbols of the
base vocabulary, we obtain:

```
("h" "u" "g", 10), ("p" "u" "g", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "u" "g" "s", 5)
```

BPE then counts the frequency of each possible symbol pair and picks the symbol pair that occurs most frequently.


key: _node_content
string_value:
{"id_": "ffa04e0f-160d-4407-8d9f-4171185ac1e2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "752258fe-7bbc-4d58-b989-b97b43430db3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "f52978d4a85a86594798709a74bd6d7e42892c06664611ff2c3051db5853e477", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5cc742e7-cf96-45ca-8b50-47d263d7afed", "node_type": "1", "metadata": {}, "hash": "a14ad6ae7ead94441aaf990b0210e8524805459183184d162385a0094aa1ff39", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7619, "end_char_idx": 9852, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
More advanced pre-tokenization include rule-based tokenization, e.g. [XLM](model_doc/xlm),
[FlauBERT](model_doc/flaubert) which uses Moses for most languages, or [GPT](model_doc/openai-gpt) which uses
spaCy and ftfy, to count the frequency of each word in the training corpus.

After pre-tokenization, a set of unique words has been created and the frequency with which each word occurred in the
training data has been determined. Next, BPE creates a base vocabulary consisting of all symbols that occur in the set
of unique words and learns merge rules to form a new symbol from two symbols of the base vocabulary. It does so until
the vocabulary has attained the desired vocabulary size. Note that the desired vocabulary size is a hyperparameter to
define before training the tokenizer.

As an example, let's assume that after pre-tokenization, the following set of words including their frequency has been
determined:

```
("hug", 10), ("pug", 5), ("pun", 12), ("bun", 4), ("hugs", 5)
```

Consequently, the base vocabulary is `["b", "g", "h", "n", "p", "s", "u"]`. Splitting all words into symbols of the
base vocabulary, we obtain:

```
("h" "u" "g", 10), ("p" "u" "g", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "u" "g" "s", 5)
```

BPE then counts the frequency of each possible symbol pair and picks the symbol pair that occurs most frequently. In
the example above `"h"` followed by `"u"` is present _10 + 5 = 15_ times (10 times in the 10 occurrences of
`"hug"`, 5 times in the 5 occurrences of `"hugs"`). However, the most frequent symbol pair is `"u"` followed by
`"g"`, occurring _10 + 5 + 5 = 20_ times in total. Thus, the first merge rule the tokenizer learns is to group all
`"u"` symbols followed by a `"g"` symbol together. Next, `"ug"` is added to the vocabulary. The set of words then
becomes

```
("h" "ug", 10), ("p" "ug", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "ug" "s", 5)
```

BPE then identifies the next most common symbol pair. It's `"u"` followed by `"n"`, which occurs 16 times. `"u"`,
`"n"` is merged to `"un"` and added to the vocabulary. The next most frequent symbol pair is `"h"` followed by
`"ug"`, occurring 15 times. Again the pair is merged and `"hug"` can be added to the vocabulary.


key: _node_content
string_value:
{"id_": "5cc742e7-cf96-45ca-8b50-47d263d7afed", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ffa04e0f-160d-4407-8d9f-4171185ac1e2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "adad7afc05d3efd318b649694e6daca16233e4fb0527b69072ef7ffaa5ef3b53", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "11299249-31f9-422d-ae4f-3015062f4ee6", "node_type": "1", "metadata": {}, "hash": "f03416eb1ee2ab84639acbb41e711ba0a065200d3dcd0f889f0354857f0fec62", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8688, "end_char_idx": 11159, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Splitting all words into symbols of the
base vocabulary, we obtain:

```
("h" "u" "g", 10), ("p" "u" "g", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "u" "g" "s", 5)
```

BPE then counts the frequency of each possible symbol pair and picks the symbol pair that occurs most frequently. In
the example above `"h"` followed by `"u"` is present _10 + 5 = 15_ times (10 times in the 10 occurrences of
`"hug"`, 5 times in the 5 occurrences of `"hugs"`). However, the most frequent symbol pair is `"u"` followed by
`"g"`, occurring _10 + 5 + 5 = 20_ times in total. Thus, the first merge rule the tokenizer learns is to group all
`"u"` symbols followed by a `"g"` symbol together. Next, `"ug"` is added to the vocabulary. The set of words then
becomes

```
("h" "ug", 10), ("p" "ug", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "ug" "s", 5)
```

BPE then identifies the next most common symbol pair. It's `"u"` followed by `"n"`, which occurs 16 times. `"u"`,
`"n"` is merged to `"un"` and added to the vocabulary. The next most frequent symbol pair is `"h"` followed by
`"ug"`, occurring 15 times. Again the pair is merged and `"hug"` can be added to the vocabulary.

At this stage, the vocabulary is `["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"]` and our set of unique words
is represented as

```
("hug", 10), ("p" "ug", 5), ("p" "un", 12), ("b" "un", 4), ("hug" "s", 5)
```

Assuming, that the Byte-Pair Encoding training would stop at this point, the learned merge rules would then be applied
to new words (as long as those new words do not include symbols that were not in the base vocabulary). For instance,
the word `"bug"` would be tokenized to `["b", "ug"]` but `"mug"` would be tokenized as `["<unk>", "ug"]` since
the symbol `"m"` is not in the base vocabulary. In general, single letters such as `"m"` are not replaced by the
`"<unk>"` symbol because the training data usually includes at least one occurrence of each letter, but it is likely
to happen for very special characters like emojis.

As mentioned earlier, the vocabulary size, *i.e.* the base vocabulary size + the number of merges, is a hyperparameter
to choose. For instance [GPT](model_doc/openai-gpt) has a vocabulary size of 40,478 since they have 478 base characters
and chose to stop training after 40,000 merges.

#### Byte-level BPE

A base vocabulary that includes all possible base characters can be quite large if *e.g.* all unicode characters are
considered as base characters.


key: _node_content
string_value:
{"id_": "11299249-31f9-422d-ae4f-3015062f4ee6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5cc742e7-cf96-45ca-8b50-47d263d7afed", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "d1f10c997b145f86030f7d8991e45f3b36bf10ecce4fe3471949928fc5ff6bfa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f2bd60c-8d54-4def-8dd4-ca8fd7a1277c", "node_type": "1", "metadata": {}, "hash": "096dc6670b28c3da7686e7d8d73d9c20b0fe65f87dc6a27c11a5a797cfc34441", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9784, "end_char_idx": 12819, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Again the pair is merged and `"hug"` can be added to the vocabulary.

At this stage, the vocabulary is `["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"]` and our set of unique words
is represented as

```
("hug", 10), ("p" "ug", 5), ("p" "un", 12), ("b" "un", 4), ("hug" "s", 5)
```

Assuming, that the Byte-Pair Encoding training would stop at this point, the learned merge rules would then be applied
to new words (as long as those new words do not include symbols that were not in the base vocabulary). For instance,
the word `"bug"` would be tokenized to `["b", "ug"]` but `"mug"` would be tokenized as `["<unk>", "ug"]` since
the symbol `"m"` is not in the base vocabulary. In general, single letters such as `"m"` are not replaced by the
`"<unk>"` symbol because the training data usually includes at least one occurrence of each letter, but it is likely
to happen for very special characters like emojis.

As mentioned earlier, the vocabulary size, *i.e.* the base vocabulary size + the number of merges, is a hyperparameter
to choose. For instance [GPT](model_doc/openai-gpt) has a vocabulary size of 40,478 since they have 478 base characters
and chose to stop training after 40,000 merges.

#### Byte-level BPE

A base vocabulary that includes all possible base characters can be quite large if *e.g.* all unicode characters are
considered as base characters. To have a better base vocabulary, [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) uses bytes
as the base vocabulary, which is a clever trick to force the base vocabulary to be of size 256 while ensuring that
every base character is included in the vocabulary. With some additional rules to deal with punctuation, the GPT2's
tokenizer can tokenize every text without the need for the <unk> symbol. [GPT-2](model_doc/gpt) has a vocabulary
size of 50,257, which corresponds to the 256 bytes base tokens, a special end-of-text token and the symbols learned
with 50,000 merges.

<a id='wordpiece'></a>

### WordPiece

WordPiece is the subword tokenization algorithm used for [BERT](model_doc/bert), [DistilBERT](model_doc/distilbert), and [Electra](model_doc/electra). The algorithm was outlined in [Japanese and Korean
Voice Search (Schuster et al., 2012)](https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf) and is very similar to
BPE. WordPiece first initializes the vocabulary to include every character present in the training data and
progressively learns a given number of merge rules. In contrast to BPE, WordPiece does not choose the most frequent
symbol pair, but the one that maximizes the likelihood of the training data once added to the vocabulary.

So what does this mean exactly? Referring to the previous example, maximizing the likelihood of the training data is
equivalent to finding the symbol pair, whose probability divided by the probabilities of its first symbol followed by
its second symbol is the greatest among all symbol pairs. *E.g.


key: _node_content
string_value:
{"id_": "6f2bd60c-8d54-4def-8dd4-ca8fd7a1277c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "11299249-31f9-422d-ae4f-3015062f4ee6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "899d2eccc2ced12bbdbca304f2c8caed6a1f4fec961684e48ff25cd22d77d606", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f95b586c-451d-4218-923b-394d25b7c830", "node_type": "1", "metadata": {}, "hash": "a199022d28c511731e84149e966ab1cece3b7efff923c849eefe3ece8bc35744", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11160, "end_char_idx": 14493, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To have a better base vocabulary, [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) uses bytes
as the base vocabulary, which is a clever trick to force the base vocabulary to be of size 256 while ensuring that
every base character is included in the vocabulary. With some additional rules to deal with punctuation, the GPT2's
tokenizer can tokenize every text without the need for the <unk> symbol. [GPT-2](model_doc/gpt) has a vocabulary
size of 50,257, which corresponds to the 256 bytes base tokens, a special end-of-text token and the symbols learned
with 50,000 merges.

<a id='wordpiece'></a>

### WordPiece

WordPiece is the subword tokenization algorithm used for [BERT](model_doc/bert), [DistilBERT](model_doc/distilbert), and [Electra](model_doc/electra). The algorithm was outlined in [Japanese and Korean
Voice Search (Schuster et al., 2012)](https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf) and is very similar to
BPE. WordPiece first initializes the vocabulary to include every character present in the training data and
progressively learns a given number of merge rules. In contrast to BPE, WordPiece does not choose the most frequent
symbol pair, but the one that maximizes the likelihood of the training data once added to the vocabulary.

So what does this mean exactly? Referring to the previous example, maximizing the likelihood of the training data is
equivalent to finding the symbol pair, whose probability divided by the probabilities of its first symbol followed by
its second symbol is the greatest among all symbol pairs. *E.g.* `"u"`, followed by `"g"` would have only been
merged if the probability of `"ug"` divided by `"u"`, `"g"` would have been greater than for any other symbol
pair. Intuitively, WordPiece is slightly different to BPE in that it evaluates what it _loses_ by merging two symbols
to ensure it's _worth it_.

<a id='unigram'></a>

### Unigram

Unigram is a subword tokenization algorithm introduced in [Subword Regularization: Improving Neural Network Translation
Models with Multiple Subword Candidates (Kudo, 2018)](https://arxiv.org/pdf/1804.10959.pdf). In contrast to BPE or
WordPiece, Unigram initializes its base vocabulary to a large number of symbols and progressively trims down each
symbol to obtain a smaller vocabulary. The base vocabulary could for instance correspond to all pre-tokenized words and
the most common substrings. Unigram is not used directly for any of the models in the transformers, but it's used in
conjunction with [SentencePiece](#sentencepiece).

At each training step, the Unigram algorithm defines a loss (often defined as the log-likelihood) over the training
data given the current vocabulary and a unigram language model. Then, for each symbol in the vocabulary, the algorithm
computes how much the overall loss would increase if the symbol was to be removed from the vocabulary. Unigram then
removes p (with p usually being 10% or 20%) percent of the symbols whose loss increase is the lowest, *i.e.* those
symbols that least affect the overall loss over the training data. This process is repeated until the vocabulary has
reached the desired size. The Unigram algorithm always keeps the base characters so that any word can be tokenized.


key: _node_content
string_value:
{"id_": "f95b586c-451d-4218-923b-394d25b7c830", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f2bd60c-8d54-4def-8dd4-ca8fd7a1277c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "1427fca741d28fbb9f7a2bd9fb37288a178cbbe71bdcca5e2e4a21af1c433d41", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ec7b47b5-e132-425d-96b6-5f15a1dae0d3", "node_type": "1", "metadata": {}, "hash": "f1e792802583c60c44ef6f9ed6afd3ea851b1f0d8c7af63f931be4901c2b39d6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12814, "end_char_idx": 15869, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*E.g.* `"u"`, followed by `"g"` would have only been
merged if the probability of `"ug"` divided by `"u"`, `"g"` would have been greater than for any other symbol
pair. Intuitively, WordPiece is slightly different to BPE in that it evaluates what it _loses_ by merging two symbols
to ensure it's _worth it_.

<a id='unigram'></a>

### Unigram

Unigram is a subword tokenization algorithm introduced in [Subword Regularization: Improving Neural Network Translation
Models with Multiple Subword Candidates (Kudo, 2018)](https://arxiv.org/pdf/1804.10959.pdf). In contrast to BPE or
WordPiece, Unigram initializes its base vocabulary to a large number of symbols and progressively trims down each
symbol to obtain a smaller vocabulary. The base vocabulary could for instance correspond to all pre-tokenized words and
the most common substrings. Unigram is not used directly for any of the models in the transformers, but it's used in
conjunction with [SentencePiece](#sentencepiece).

At each training step, the Unigram algorithm defines a loss (often defined as the log-likelihood) over the training
data given the current vocabulary and a unigram language model. Then, for each symbol in the vocabulary, the algorithm
computes how much the overall loss would increase if the symbol was to be removed from the vocabulary. Unigram then
removes p (with p usually being 10% or 20%) percent of the symbols whose loss increase is the lowest, *i.e.* those
symbols that least affect the overall loss over the training data. This process is repeated until the vocabulary has
reached the desired size. The Unigram algorithm always keeps the base characters so that any word can be tokenized.

Because Unigram is not based on merge rules (in contrast to BPE and WordPiece), the algorithm has several ways of
tokenizing new text after training. As an example, if a trained Unigram tokenizer exhibits the vocabulary:

```
["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"],
```

`"hugs"` could be tokenized both as `["hug", "s"]`, `["h", "ug", "s"]` or `["h", "u", "g", "s"]`. So which one
to choose? Unigram saves the probability of each token in the training corpus on top of saving the vocabulary so that
the probability of each possible tokenization can be computed after training. The algorithm simply picks the most
likely tokenization in practice, but also offers the possibility to sample a possible tokenization according to their
probabilities.

Those probabilities are defined by the loss the tokenizer is trained on. Assuming that the training data consists of
the words \\(x_{1}, \dots, x_{N}\\) and that the set of all possible tokenizations for a word \\(x_{i}\\) is
defined as \\(S(x_{i})\\), then the overall loss is defined as

$$\mathcal{L} = -\sum_{i=1}^{N} \log \left ( \sum_{x \in S(x_{i})} p(x) \right )$$

<a id='sentencepiece'></a>

### SentencePiece

All tokenization algorithms described so far have the same problem: It is assumed that the input text uses spaces to
separate words. However, not all languages use spaces to separate words.


key: _node_content
string_value:
{"id_": "ec7b47b5-e132-425d-96b6-5f15a1dae0d3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af1796f7-3a48-4d41-af2e-6bd46e56ed94", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f95b586c-451d-4218-923b-394d25b7c830", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers"}, "hash": "6e6609527a2ac9f496a884fce72d1bb3ecb47b199d9e36befb2b06197f9e2b9a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14404, "end_char_idx": 16925, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Unigram algorithm always keeps the base characters so that any word can be tokenized.

Because Unigram is not based on merge rules (in contrast to BPE and WordPiece), the algorithm has several ways of
tokenizing new text after training. As an example, if a trained Unigram tokenizer exhibits the vocabulary:

```
["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"],
```

`"hugs"` could be tokenized both as `["hug", "s"]`, `["h", "ug", "s"]` or `["h", "u", "g", "s"]`. So which one
to choose? Unigram saves the probability of each token in the training corpus on top of saving the vocabulary so that
the probability of each possible tokenization can be computed after training. The algorithm simply picks the most
likely tokenization in practice, but also offers the possibility to sample a possible tokenization according to their
probabilities.

Those probabilities are defined by the loss the tokenizer is trained on. Assuming that the training data consists of
the words \\(x_{1}, \dots, x_{N}\\) and that the set of all possible tokenizations for a word \\(x_{i}\\) is
defined as \\(S(x_{i})\\), then the overall loss is defined as

$$\mathcal{L} = -\sum_{i=1}^{N} \log \left ( \sum_{x \in S(x_{i})} p(x) \right )$$

<a id='sentencepiece'></a>

### SentencePiece

All tokenization algorithms described so far have the same problem: It is assumed that the input text uses spaces to
separate words. However, not all languages use spaces to separate words. One possible solution is to use language
specific pre-tokenizers, *e.g.* [XLM](model_doc/xlm) uses a specific Chinese, Japanese, and Thai pre-tokenizer.
To solve this problem more generally, [SentencePiece: A simple and language independent subword tokenizer and
detokenizer for Neural Text Processing (Kudo et al., 2018)](https://arxiv.org/pdf/1808.06226.pdf) treats the input
as a raw input stream, thus including the space in the set of characters to use. It then uses the BPE or unigram
algorithm to construct the appropriate vocabulary.

The [`XLNetTokenizer`] uses SentencePiece for example, which is also why in the example earlier the
`"▁"` character was included in the vocabulary. Decoding with SentencePiece is very easy since all tokens can just be
concatenated and `"▁"` is replaced by a space.

All transformers models in the library that use SentencePiece use it in combination with unigram. Examples of models
using SentencePiece are [ALBERT](model_doc/albert), [XLNet](model_doc/xlnet), [Marian](model_doc/marian), and [T5](model_doc/t5).


key: _node_content
string_value:
{"id_": "33cba0e8-d3da-42fb-9e25-30c0c636fba8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3ba0e201-a345-47a4-b8ea-99ad353957b3", "node_type": "1", "metadata": {}, "hash": "b259ba8d0f4db412c3051f2fe4331a9609ef34d1d9270717ff5e2bc90f5991a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2883, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DeepSpeed

[DeepSpeed](https://www.deepspeed.ai/) is a PyTorch optimization library that makes distributed training memory-efficient and fast. At its core is the [Zero Redundancy Optimizer (ZeRO)](https://hf.co/papers/1910.02054) which enables training large models at scale. ZeRO works in several stages:

* ZeRO-1, optimizer state partitioning across GPUs
* ZeRO-2, gradient partitioning across GPUs
* ZeRO-3, parameter partitioning across GPUs

In GPU-limited environments, ZeRO also enables offloading optimizer memory and computation from the GPU to the CPU to fit and train really large models on a single GPU. DeepSpeed is integrated with the Transformers [`Trainer`] class for all ZeRO stages and offloading. All you need to do is provide a config file or you can use a provided template. For inference, Transformers support ZeRO-3 and offloading since it allows loading huge models.

This guide will walk you through how to deploy DeepSpeed training, the features you can enable, how to setup the config files for different ZeRO stages, offloading, inference, and using DeepSpeed without the [`Trainer`].

## Installation

DeepSpeed is available to install from PyPI or Transformers (for more detailed installation options, take a look at the DeepSpeed [installation details](https://www.deepspeed.ai/tutorials/advanced-install/) or the GitHub [README](https://github.com/microsoft/deepspeed#installation)).

<Tip>

If you're having difficulties installing DeepSpeed, check the [DeepSpeed CUDA installation](../debugging#deepspeed-cuda-installation) guide. While DeepSpeed has a pip installable PyPI package, it is highly recommended to [install it from source](https://www.deepspeed.ai/tutorials/advanced-install/#install-deepspeed-from-source) to best match your hardware and to support certain features, like 1-bit Adam, which aren’t available in the PyPI distribution.

</Tip>

<hfoptions id="install">
<hfoption id="PyPI">

```bash
pip install deepspeed
```

</hfoption>
<hfoption id="Transformers">

```bash
pip install transformers[deepspeed]
```

</hfoption>
</hfoptions>

## Memory requirements

Before you begin, it is a good idea to check whether you have enough GPU and CPU memory to fit your model. DeepSpeed provides a tool for estimating the required CPU/GPU memory. For example, to estimate the memory requirements for the [bigscience/T0_3B](bigscience/T0_3B) model on a single GPU:

```bash
$ python -c 'from transformers import AutoModel; \
from deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live; \
model = AutoModel.from_pretrained("bigscience/T0_3B"); \
estimate_zero3_model_states_mem_needs_all_live(model, num_gpus_per_node=1, num_nodes=1)'
[...]
Estimated memory needed for params, optim states and gradients for a:
HW: Setup with 1 node, 1 GPU per node.
SW: Model with 2783M total params, 65M largest layer params.


key: _node_content
string_value:
{"id_": "3ba0e201-a345-47a4-b8ea-99ad353957b3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "33cba0e8-d3da-42fb-9e25-30c0c636fba8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "99f2398d62a4251882d0fc1e5a1038425db3984644438f7c0b6612876dbd112c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a05eee3-763c-4f3b-aca8-c96d08ecec9a", "node_type": "1", "metadata": {}, "hash": "d426d2dbab6d384d3da2fd4eaae004d80281ba432f770744071b926c91a644de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1420, "end_char_idx": 4081, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

If you're having difficulties installing DeepSpeed, check the [DeepSpeed CUDA installation](../debugging#deepspeed-cuda-installation) guide. While DeepSpeed has a pip installable PyPI package, it is highly recommended to [install it from source](https://www.deepspeed.ai/tutorials/advanced-install/#install-deepspeed-from-source) to best match your hardware and to support certain features, like 1-bit Adam, which aren’t available in the PyPI distribution.

</Tip>

<hfoptions id="install">
<hfoption id="PyPI">

```bash
pip install deepspeed
```

</hfoption>
<hfoption id="Transformers">

```bash
pip install transformers[deepspeed]
```

</hfoption>
</hfoptions>

## Memory requirements

Before you begin, it is a good idea to check whether you have enough GPU and CPU memory to fit your model. DeepSpeed provides a tool for estimating the required CPU/GPU memory. For example, to estimate the memory requirements for the [bigscience/T0_3B](bigscience/T0_3B) model on a single GPU:

```bash
$ python -c 'from transformers import AutoModel; \
from deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live; \
model = AutoModel.from_pretrained("bigscience/T0_3B"); \
estimate_zero3_model_states_mem_needs_all_live(model, num_gpus_per_node=1, num_nodes=1)'
[...]
Estimated memory needed for params, optim states and gradients for a:
HW: Setup with 1 node, 1 GPU per node.
SW: Model with 2783M total params, 65M largest layer params.
  per CPU  |  per GPU |   Options
   70.00GB |   0.25GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1
   70.00GB |   0.25GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0
   62.23GB |   5.43GB | offload_param=none, offload_optimizer=cpu , zero_init=1
   62.23GB |   5.43GB | offload_param=none, offload_optimizer=cpu , zero_init=0
    0.37GB |  46.91GB | offload_param=none, offload_optimizer=none, zero_init=1
   15.56GB |  46.91GB | offload_param=none, offload_optimizer=none, zero_init=0
```

This means you either need a single 80GB GPU without CPU offload or a 8GB GPU and a ~60GB CPU to offload to (these are just the memory requirements for the parameters, optimizer states and gradients, and you'll need a bit more for the CUDA kernels and activations). You should also consider the tradeoff between cost and speed because it'll be cheaper to rent or buy a smaller GPU but it'll take longer to train your model.

If you have enough GPU memory make sure you disable CPU/NVMe offload to make everything faster.

## Select a ZeRO stage

After you've installed DeepSpeed and have a better idea of your memory requirements, the next step is selecting a ZeRO stage to use.


key: _node_content
string_value:
{"id_": "0a05eee3-763c-4f3b-aca8-c96d08ecec9a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3ba0e201-a345-47a4-b8ea-99ad353957b3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "e1ff953fc47b9cd37de0e3fde8f8612cf634650fd12bc09c7863e1d46d8c783d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49d8e529-8fbf-442d-8f18-7522df2ce534", "node_type": "1", "metadata": {}, "hash": "20474397d829d08f4ff0fa535341607a156dc33dc7635e05ebd4f86163aa1664", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2823, "end_char_idx": 5722, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
SW: Model with 2783M total params, 65M largest layer params.
  per CPU  |  per GPU |   Options
   70.00GB |   0.25GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1
   70.00GB |   0.25GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0
   62.23GB |   5.43GB | offload_param=none, offload_optimizer=cpu , zero_init=1
   62.23GB |   5.43GB | offload_param=none, offload_optimizer=cpu , zero_init=0
    0.37GB |  46.91GB | offload_param=none, offload_optimizer=none, zero_init=1
   15.56GB |  46.91GB | offload_param=none, offload_optimizer=none, zero_init=0
```

This means you either need a single 80GB GPU without CPU offload or a 8GB GPU and a ~60GB CPU to offload to (these are just the memory requirements for the parameters, optimizer states and gradients, and you'll need a bit more for the CUDA kernels and activations). You should also consider the tradeoff between cost and speed because it'll be cheaper to rent or buy a smaller GPU but it'll take longer to train your model.

If you have enough GPU memory make sure you disable CPU/NVMe offload to make everything faster.

## Select a ZeRO stage

After you've installed DeepSpeed and have a better idea of your memory requirements, the next step is selecting a ZeRO stage to use. In order of fastest and most memory-efficient:

| Fastest          | Memory efficient |
|------------------|------------------|
| ZeRO-1           | ZeRO-3 + offload |
| ZeRO-2           | ZeRO-3           |
| ZeRO-2 + offload | ZeRO-2 + offload |
| ZeRO-3           | ZeRO-2           |
| ZeRO-3 + offload | ZeRO-1           |

To find what works best for you, start with the fastest approach and if you run out of memory, try the next stage which is slower but more memory efficient. Feel free to work in whichever direction you prefer (starting with the most memory efficient or fastest) to discover the appropriate balance between speed and memory usage.

A general process you can use is (start with batch size of 1):

1. enable gradient checkpointing
2. try ZeRO-2
3. try ZeRO-2 and offload the optimizer
4. try ZeRO-3
5. try ZeRO-3 and offload parameters to the CPU
6. try ZeRO-3 and offload parameters and the optimizer to the CPU
7. try lowering various default values like a narrower search beam if you're using the [`~GenerationMixin.generate`] method
8. try mixed half-precision (fp16 on older GPU architectures and bf16 on Ampere) over full-precision weights
9. add more hardware if possible or enable Infinity to offload parameters and the optimizer to a NVMe
10. once you're not running out of memory, measure effective throughput and then try to increase the batch size as large as you can to maximize GPU efficiency
11. lastly, try to optimize your training setup by disabling some offload features or use a faster ZeRO stage and increasing/decreasing the batch size to find the best tradeoff between speed and memory usage


key: _node_content
string_value:
{"id_": "49d8e529-8fbf-442d-8f18-7522df2ce534", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a05eee3-763c-4f3b-aca8-c96d08ecec9a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "90629852709f297cbd0a8aed89b2f74e4d7969cadb9c74a6ff6873e794f5a83a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "05bc3d91-2bf7-447c-873c-dbae135df00c", "node_type": "1", "metadata": {}, "hash": "57c662e2d5b645555a19baa75f67d852ec634028cd166a6c796b28d19e0a6a19", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4082, "end_char_idx": 7404, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In order of fastest and most memory-efficient:

| Fastest          | Memory efficient |
|------------------|------------------|
| ZeRO-1           | ZeRO-3 + offload |
| ZeRO-2           | ZeRO-3           |
| ZeRO-2 + offload | ZeRO-2 + offload |
| ZeRO-3           | ZeRO-2           |
| ZeRO-3 + offload | ZeRO-1           |

To find what works best for you, start with the fastest approach and if you run out of memory, try the next stage which is slower but more memory efficient. Feel free to work in whichever direction you prefer (starting with the most memory efficient or fastest) to discover the appropriate balance between speed and memory usage.

A general process you can use is (start with batch size of 1):

1. enable gradient checkpointing
2. try ZeRO-2
3. try ZeRO-2 and offload the optimizer
4. try ZeRO-3
5. try ZeRO-3 and offload parameters to the CPU
6. try ZeRO-3 and offload parameters and the optimizer to the CPU
7. try lowering various default values like a narrower search beam if you're using the [`~GenerationMixin.generate`] method
8. try mixed half-precision (fp16 on older GPU architectures and bf16 on Ampere) over full-precision weights
9. add more hardware if possible or enable Infinity to offload parameters and the optimizer to a NVMe
10. once you're not running out of memory, measure effective throughput and then try to increase the batch size as large as you can to maximize GPU efficiency
11. lastly, try to optimize your training setup by disabling some offload features or use a faster ZeRO stage and increasing/decreasing the batch size to find the best tradeoff between speed and memory usage


## DeepSpeed configuration file

DeepSpeed works with the [`Trainer`] class by way of a config file containing all the parameters for configuring how you want setup your training run. When you execute your training script, DeepSpeed logs the configuration it received from [`Trainer`] to the console so you can see exactly what configuration was used.

<Tip>

Find a complete list of DeepSpeed configuration options on the [DeepSpeed Configuration JSON](https://www.deepspeed.ai/docs/config-json/) reference. You can also find more practical examples of various DeepSpeed configuration examples on the [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples) repository or the main [DeepSpeed](https://github.com/microsoft/DeepSpeed) repository. To quickly find specific examples, you can:

```bash
git clone https://github.com/microsoft/DeepSpeedExamples
cd DeepSpeedExamples
find . -name '*json'
# find examples with the Lamb optimizer
grep -i Lamb $(find . -name '*json')
```

</Tip>

The DeepSpeed configuration file is passed as a path to a JSON file if you're training from the command line interface or as a nested `dict` object if you're using the [`Trainer`] in a notebook setting.

<hfoptions id="pass-config">
<hfoption id="path to file">

```py
TrainingArguments(..., deepspeed="path/to/deepspeed_config.json")
```

</hfoption>
<hfoption id="nested dict">

```py
ds_config_dict = dict(scheduler=scheduler_params, optimizer=optimizer_params)
args = TrainingArguments(..., deepspeed=ds_config_dict)
trainer = Trainer(model, args, ...)
```

</hfoption>
</hfoptions>

### DeepSpeed and Trainer parameters

There are three types of configuration parameters:

1.


key: _node_content
string_value:
{"id_": "05bc3d91-2bf7-447c-873c-dbae135df00c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "49d8e529-8fbf-442d-8f18-7522df2ce534", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "6ca4d0ae303ef0e2dda560f5089ba14f7745228a4a48a8ab94ff402980c92e8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7339fa0f-2265-4406-b650-7532778867b4", "node_type": "1", "metadata": {}, "hash": "6a484ff399c53116700d6212e3896df029a3b4418281542312ec2188cd9fe06b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5725, "end_char_idx": 9333, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## DeepSpeed configuration file

DeepSpeed works with the [`Trainer`] class by way of a config file containing all the parameters for configuring how you want setup your training run. When you execute your training script, DeepSpeed logs the configuration it received from [`Trainer`] to the console so you can see exactly what configuration was used.

<Tip>

Find a complete list of DeepSpeed configuration options on the [DeepSpeed Configuration JSON](https://www.deepspeed.ai/docs/config-json/) reference. You can also find more practical examples of various DeepSpeed configuration examples on the [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples) repository or the main [DeepSpeed](https://github.com/microsoft/DeepSpeed) repository. To quickly find specific examples, you can:

```bash
git clone https://github.com/microsoft/DeepSpeedExamples
cd DeepSpeedExamples
find . -name '*json'
# find examples with the Lamb optimizer
grep -i Lamb $(find . -name '*json')
```

</Tip>

The DeepSpeed configuration file is passed as a path to a JSON file if you're training from the command line interface or as a nested `dict` object if you're using the [`Trainer`] in a notebook setting.

<hfoptions id="pass-config">
<hfoption id="path to file">

```py
TrainingArguments(..., deepspeed="path/to/deepspeed_config.json")
```

</hfoption>
<hfoption id="nested dict">

```py
ds_config_dict = dict(scheduler=scheduler_params, optimizer=optimizer_params)
args = TrainingArguments(..., deepspeed=ds_config_dict)
trainer = Trainer(model, args, ...)
```

</hfoption>
</hfoptions>

### DeepSpeed and Trainer parameters

There are three types of configuration parameters:

1. Some of the configuration parameters are shared by [`Trainer`] and DeepSpeed, and it can be difficult to identify errors when there are conflicting definitions. To make it easier, these shared configuration parameters are configured from the [`Trainer`] command line arguments.

2. Some configuration parameters that are automatically derived from the model configuration so you don't need to manually adjust these values. The [`Trainer`] uses a configuration value `auto` to determine set the most correct or efficient value. You could set your own configuration parameters explicitly, but you must take care to ensure the [`Trainer`] arguments and DeepSpeed configuration parameters agree. Mismatches may cause the training to fail in very difficult to detect ways!

3. Some configuration parameters specific to DeepSpeed only which need to be manually set based on your training needs.

You could also modify the DeepSpeed configuration and edit [`TrainingArguments`] from it:

1. Create or load a DeepSpeed configuration to use as the main configuration
2. Create a [`TrainingArguments`] object based on these DeepSpeed configuration values

Some values, such as `scheduler.params.total_num_steps` are calculated by the [`Trainer`] during training.

### ZeRO configuration

There are three configurations, each corresponding to a different ZeRO stage. Stage 1 is not as interesting for scalability, and this guide focuses on stages 2 and 3. The `zero_optimization` configuration contains all the options for what to enable and how to configure them. For a more detailed explanation of each parameter, take a look at the [DeepSpeed Configuration JSON](https://www.deepspeed.ai/docs/config-json/) reference.

<Tip warning={true}>
DeepSpeed doesn’t validate parameter names and any typos fallback on the parameter's default setting. You can watch the DeepSpeed engine startup log messages to see what values it is going to use.


key: _node_content
string_value:
{"id_": "7339fa0f-2265-4406-b650-7532778867b4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "05bc3d91-2bf7-447c-873c-dbae135df00c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "e739c4eb4f0c22280dfd5a4d30e815864797426704e21be4ee45741a2274357f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7541adfc-381b-4b2b-a6d0-80ee8dcbe1c4", "node_type": "1", "metadata": {}, "hash": "09dcc7ff68e9a762c1dc1aecd14cada7bd615151246ce5cb7319a6215f1b900e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7405, "end_char_idx": 10946, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Some of the configuration parameters are shared by [`Trainer`] and DeepSpeed, and it can be difficult to identify errors when there are conflicting definitions. To make it easier, these shared configuration parameters are configured from the [`Trainer`] command line arguments.

2. Some configuration parameters that are automatically derived from the model configuration so you don't need to manually adjust these values. The [`Trainer`] uses a configuration value `auto` to determine set the most correct or efficient value. You could set your own configuration parameters explicitly, but you must take care to ensure the [`Trainer`] arguments and DeepSpeed configuration parameters agree. Mismatches may cause the training to fail in very difficult to detect ways!

3. Some configuration parameters specific to DeepSpeed only which need to be manually set based on your training needs.

You could also modify the DeepSpeed configuration and edit [`TrainingArguments`] from it:

1. Create or load a DeepSpeed configuration to use as the main configuration
2. Create a [`TrainingArguments`] object based on these DeepSpeed configuration values

Some values, such as `scheduler.params.total_num_steps` are calculated by the [`Trainer`] during training.

### ZeRO configuration

There are three configurations, each corresponding to a different ZeRO stage. Stage 1 is not as interesting for scalability, and this guide focuses on stages 2 and 3. The `zero_optimization` configuration contains all the options for what to enable and how to configure them. For a more detailed explanation of each parameter, take a look at the [DeepSpeed Configuration JSON](https://www.deepspeed.ai/docs/config-json/) reference.

<Tip warning={true}>
DeepSpeed doesn’t validate parameter names and any typos fallback on the parameter's default setting. You can watch the DeepSpeed engine startup log messages to see what values it is going to use.

</Tip>

The following configurations must be setup with DeepSpeed because the [`Trainer`] doesn't provide equivalent command line arguments.

<hfoptions id="zero-config">
<hfoption id="ZeRO-1">

ZeRO-1 shards the optimizer states across GPUs, and you can expect a tiny speed up. The ZeRO-1 config can be setup like this:

```yml
{
    "zero_optimization": {
        "stage": 1
    }
}
```

</hfoption>
<hfoption id="ZeRO-2">

ZeRO-2 shards the optimizer and gradients across GPUs. This stage is primarily used for training since its features are not relevant to inference. Some important parameters to configure for better performance include:

* `offload_optimizer` should be enabled to reduce GPU memory usage.
* `overlap_comm` when set to `true` trades off increased GPU memory usage to lower allreduce latency. This feature uses 4.5x the `allgather_bucket_size` and `reduce_bucket_size` values. In this example, they're set to `5e8` which means it requires 9GB of GPU memory. If your GPU memory is 8GB or less, you should reduce `overlap_comm` to lower the memory requirements and prevent an out-of-memory (OOM) error.
* `allgather_bucket_size` and `reduce_bucket_size` trade off available GPU memory for communication speed. The smaller their values, the slower communication is and the more GPU memory is available. You can balance, for example, whether a bigger batch size is more important than a slightly slower training time.
* `round_robin_gradients` is available in DeepSpeed 0.4.4 for CPU offloading. It parallelizes gradient copying to CPU memory among ranks by fine-grained gradient partitioning.


key: _node_content
string_value:
{"id_": "7541adfc-381b-4b2b-a6d0-80ee8dcbe1c4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7339fa0f-2265-4406-b650-7532778867b4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "6764a321140e85053a029f6dc3618cc864bab791e5fa49c16db4df2501b77e03", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d29e531a-e330-45db-b030-50092ce1e6f3", "node_type": "1", "metadata": {}, "hash": "e622bce42177675c9ae72b30ef5c8a69e52438001d9a446e785044d319815e6e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9335, "end_char_idx": 12545, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

The following configurations must be setup with DeepSpeed because the [`Trainer`] doesn't provide equivalent command line arguments.

<hfoptions id="zero-config">
<hfoption id="ZeRO-1">

ZeRO-1 shards the optimizer states across GPUs, and you can expect a tiny speed up. The ZeRO-1 config can be setup like this:

```yml
{
    "zero_optimization": {
        "stage": 1
    }
}
```

</hfoption>
<hfoption id="ZeRO-2">

ZeRO-2 shards the optimizer and gradients across GPUs. This stage is primarily used for training since its features are not relevant to inference. Some important parameters to configure for better performance include:

* `offload_optimizer` should be enabled to reduce GPU memory usage.
* `overlap_comm` when set to `true` trades off increased GPU memory usage to lower allreduce latency. This feature uses 4.5x the `allgather_bucket_size` and `reduce_bucket_size` values. In this example, they're set to `5e8` which means it requires 9GB of GPU memory. If your GPU memory is 8GB or less, you should reduce `overlap_comm` to lower the memory requirements and prevent an out-of-memory (OOM) error.
* `allgather_bucket_size` and `reduce_bucket_size` trade off available GPU memory for communication speed. The smaller their values, the slower communication is and the more GPU memory is available. You can balance, for example, whether a bigger batch size is more important than a slightly slower training time.
* `round_robin_gradients` is available in DeepSpeed 0.4.4 for CPU offloading. It parallelizes gradient copying to CPU memory among ranks by fine-grained gradient partitioning. Performance benefit grows with gradient accumulation steps (more copying between optimizer steps) or GPU count (increased parallelism).

```yml
{
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 5e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": 5e8,
        "contiguous_gradients": true
        "round_robin_gradients": true
    }
}
```

</hfoption>
<hfoption id="ZeRO-3">

ZeRO-3 shards the optimizer, gradient, and parameters across GPUs. Unlike ZeRO-2, ZeRO-3 can also be used for inference, in addition to training, because it allows large models to be loaded on multiple GPUs. Some important parameters to configure include:

* `device: "cpu"` can help if you're running out of GPU memory and if you have free CPU memory available. This allows offloading model parameters to the CPU.
* `pin_memory: true` can improve throughput, but less memory becomes available for other processes because the pinned memory is reserved for the specific process that requested it and it's typically accessed much faster than normal CPU memory.
* `stage3_max_live_parameters` is the upper limit on how many full parameters you want to keep on the GPU at any given time. Reduce this value if you encounter an OOM error.
* `stage3_max_reuse_distance` is a value for determining when a parameter is used again in the future, and it helps decide whether to throw the parameter away or to keep it.


key: _node_content
string_value:
{"id_": "d29e531a-e330-45db-b030-50092ce1e6f3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7541adfc-381b-4b2b-a6d0-80ee8dcbe1c4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "c27551aa5dcc5030086f0692570a26076f145a8da423f5696ed02be7032fe34e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "24ac8af7-b836-4e25-bc87-6fe5e9573f9e", "node_type": "1", "metadata": {}, "hash": "1fb4afe71567f903c105a1c70ed704cb3a5d504d30934b92f1179b12dc5d91c1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10849, "end_char_idx": 14209, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It parallelizes gradient copying to CPU memory among ranks by fine-grained gradient partitioning. Performance benefit grows with gradient accumulation steps (more copying between optimizer steps) or GPU count (increased parallelism).

```yml
{
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 5e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": 5e8,
        "contiguous_gradients": true
        "round_robin_gradients": true
    }
}
```

</hfoption>
<hfoption id="ZeRO-3">

ZeRO-3 shards the optimizer, gradient, and parameters across GPUs. Unlike ZeRO-2, ZeRO-3 can also be used for inference, in addition to training, because it allows large models to be loaded on multiple GPUs. Some important parameters to configure include:

* `device: "cpu"` can help if you're running out of GPU memory and if you have free CPU memory available. This allows offloading model parameters to the CPU.
* `pin_memory: true` can improve throughput, but less memory becomes available for other processes because the pinned memory is reserved for the specific process that requested it and it's typically accessed much faster than normal CPU memory.
* `stage3_max_live_parameters` is the upper limit on how many full parameters you want to keep on the GPU at any given time. Reduce this value if you encounter an OOM error.
* `stage3_max_reuse_distance` is a value for determining when a parameter is used again in the future, and it helps decide whether to throw the parameter away or to keep it. If the parameter is going to be reused (if the value is less than `stage3_max_reuse_distance`), then it is kept to reduce communication overhead. This is super helpful when activation checkpointing is enabled and you want to keep the parameter in the forward recompute until the backward pass. But reduce this value if you encounter an OOM error.
* `stage3_gather_16bit_weights_on_model_save` consolidates fp16 weights when a model is saved. For large models and multiple GPUs, this is expensive in terms of memory and speed. You should enable it if you're planning on resuming training.
* `sub_group_size` controls which parameters are updated during the optimizer step. Parameters are grouped into buckets of `sub_group_size` and each bucket is updated one at a time. When used with NVMe offload, `sub_group_size` determines when model states are moved in and out of CPU memory from during the optimization step. This prevents running out of CPU memory for extremely large models. `sub_group_size` can be left to its default value if you aren't using NVMe offload, but you may want to change it if you:

    1. Run into an OOM error during the optimizer step. In this case, reduce `sub_group_size` to reduce memory usage of the temporary buffers.
    2. The optimizer step is taking a really long time. In this case, increase `sub_group_size` to improve bandwidth utilization as a result of increased data buffers.

* `reduce_bucket_size`, `stage3_prefetch_bucket_size`, and `stage3_param_persistence_threshold` are dependent on a model's hidden size. It is recommended to set these values to `auto` and allow the [`Trainer`] to automatically assign the values.


key: _node_content
string_value:
{"id_": "24ac8af7-b836-4e25-bc87-6fe5e9573f9e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d29e531a-e330-45db-b030-50092ce1e6f3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "40f9c95409c9c7bcd33c26916522ffe38891ec0223f57625f60553ece1e129d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "548af367-885c-4295-89d2-67dce1db5938", "node_type": "1", "metadata": {}, "hash": "4cecf0cf090bf952d33b283043aaeb74f4e5134be7d21b202cfb1b35592e6e66", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12546, "end_char_idx": 15535, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If the parameter is going to be reused (if the value is less than `stage3_max_reuse_distance`), then it is kept to reduce communication overhead. This is super helpful when activation checkpointing is enabled and you want to keep the parameter in the forward recompute until the backward pass. But reduce this value if you encounter an OOM error.
* `stage3_gather_16bit_weights_on_model_save` consolidates fp16 weights when a model is saved. For large models and multiple GPUs, this is expensive in terms of memory and speed. You should enable it if you're planning on resuming training.
* `sub_group_size` controls which parameters are updated during the optimizer step. Parameters are grouped into buckets of `sub_group_size` and each bucket is updated one at a time. When used with NVMe offload, `sub_group_size` determines when model states are moved in and out of CPU memory from during the optimization step. This prevents running out of CPU memory for extremely large models. `sub_group_size` can be left to its default value if you aren't using NVMe offload, but you may want to change it if you:

    1. Run into an OOM error during the optimizer step. In this case, reduce `sub_group_size` to reduce memory usage of the temporary buffers.
    2. The optimizer step is taking a really long time. In this case, increase `sub_group_size` to improve bandwidth utilization as a result of increased data buffers.

* `reduce_bucket_size`, `stage3_prefetch_bucket_size`, and `stage3_param_persistence_threshold` are dependent on a model's hidden size. It is recommended to set these values to `auto` and allow the [`Trainer`] to automatically assign the values.

```yml
{
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_16bit_weights_on_model_save": true
    }
}
```

You can use the [`deepspeed.zero.Init`](https://deepspeed.readthedocs.io/en/latest/zero3.html#deepspeed.zero.Init) context manager to initialize a model faster:

```py
from transformers import T5ForConditionalGeneration, T5Config
import deepspeed

with deepspeed.zero.Init():
    config = T5Config.from_pretrained("google-t5/t5-small")
    model = T5ForConditionalGeneration(config)
```

For pretrained models, the DeepSped config file needs to have `is_deepspeed_zero3_enabled: true` setup in [`TrainingArguments`] and it needs a ZeRO configuration enabled. The [`TrainingArguments`] object must be created **before** calling the model [`~PreTrainedModel.from_pretrained`].


key: _node_content
string_value:
{"id_": "548af367-885c-4295-89d2-67dce1db5938", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "24ac8af7-b836-4e25-bc87-6fe5e9573f9e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "9bc21fcaf586badc8cde3d6fee053814b8e3bd22dcdd0ab1645b6daa6f15369d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b804e41c-c024-415e-a237-d023e8522938", "node_type": "1", "metadata": {}, "hash": "25f10ff421ad9688a69ac27af21316983380f40eee7aa361b57494631fa64da0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13964, "end_char_idx": 17184, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* `reduce_bucket_size`, `stage3_prefetch_bucket_size`, and `stage3_param_persistence_threshold` are dependent on a model's hidden size. It is recommended to set these values to `auto` and allow the [`Trainer`] to automatically assign the values.

```yml
{
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_16bit_weights_on_model_save": true
    }
}
```

You can use the [`deepspeed.zero.Init`](https://deepspeed.readthedocs.io/en/latest/zero3.html#deepspeed.zero.Init) context manager to initialize a model faster:

```py
from transformers import T5ForConditionalGeneration, T5Config
import deepspeed

with deepspeed.zero.Init():
    config = T5Config.from_pretrained("google-t5/t5-small")
    model = T5ForConditionalGeneration(config)
```

For pretrained models, the DeepSped config file needs to have `is_deepspeed_zero3_enabled: true` setup in [`TrainingArguments`] and it needs a ZeRO configuration enabled. The [`TrainingArguments`] object must be created **before** calling the model [`~PreTrainedModel.from_pretrained`].

```py
from transformers import AutoModel, Trainer, TrainingArguments

training_args = TrainingArguments(..., deepspeed=ds_config)
model = AutoModel.from_pretrained("google-t5/t5-small")
trainer = Trainer(model=model, args=training_args, ...)
```

You'll need ZeRO-3 if the fp16 weights don't fit on a single GPU. If you're able to load fp16 weights, then make sure you specify `torch_dtype=torch.float16` in [`~PreTrainedModel.from_pretrained`].

Another consideration for ZeRO-3 is if you have multiple GPUs, no single GPU has all the parameters unless it's the parameters for the currently executing layer. To access all parameters from all the layers at once, such as loading pretrained model weights in [`~PreTrainedModel.from_pretrained`], one layer is loaded at a time and immediately partitioned to all GPUs. This is because for very large models, it isn't possible to load the weights on one GPU and then distribute them across the other GPUs due to memory limitations.

If you encounter a model parameter weight that looks like the following, where `tensor([1.])` or the parameter size is 1 instead of a larger multi-dimensional shape, this means the parameter is partitioned and this is a ZeRO-3 placeholder.

```py
tensor([1.0], device="cuda:0", dtype=torch.float16, requires_grad=True)
```

<Tip>

For more information about initializing large models with ZeRO-3 and accessing the parameters, take a look at the [Constructing Massive Models](https://deepspeed.readthedocs.io/en/latest/zero3.html#constructing-massive-models) and [Gathering Parameters](https://deepspeed.readthedocs.io/en/latest/zero3.html#gathering-parameters) guides.


key: _node_content
string_value:
{"id_": "b804e41c-c024-415e-a237-d023e8522938", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "548af367-885c-4295-89d2-67dce1db5938", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "2beb106c23bd018843c574be1d457c6080a8e330c49d016c108f76e6d1b129f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d4b78d2f-9e69-4c89-bedc-f48627bd9d76", "node_type": "1", "metadata": {}, "hash": "bae6a1593270c09387577b7670ea77ab58a0fb31389f96de838864646d942c9a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15537, "end_char_idx": 18485, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import AutoModel, Trainer, TrainingArguments

training_args = TrainingArguments(..., deepspeed=ds_config)
model = AutoModel.from_pretrained("google-t5/t5-small")
trainer = Trainer(model=model, args=training_args, ...)
```

You'll need ZeRO-3 if the fp16 weights don't fit on a single GPU. If you're able to load fp16 weights, then make sure you specify `torch_dtype=torch.float16` in [`~PreTrainedModel.from_pretrained`].

Another consideration for ZeRO-3 is if you have multiple GPUs, no single GPU has all the parameters unless it's the parameters for the currently executing layer. To access all parameters from all the layers at once, such as loading pretrained model weights in [`~PreTrainedModel.from_pretrained`], one layer is loaded at a time and immediately partitioned to all GPUs. This is because for very large models, it isn't possible to load the weights on one GPU and then distribute them across the other GPUs due to memory limitations.

If you encounter a model parameter weight that looks like the following, where `tensor([1.])` or the parameter size is 1 instead of a larger multi-dimensional shape, this means the parameter is partitioned and this is a ZeRO-3 placeholder.

```py
tensor([1.0], device="cuda:0", dtype=torch.float16, requires_grad=True)
```

<Tip>

For more information about initializing large models with ZeRO-3 and accessing the parameters, take a look at the [Constructing Massive Models](https://deepspeed.readthedocs.io/en/latest/zero3.html#constructing-massive-models) and [Gathering Parameters](https://deepspeed.readthedocs.io/en/latest/zero3.html#gathering-parameters) guides.

</Tip>

</hfoption>
</hfoptions>

### NVMe configuration

[ZeRO-Infinity](https://hf.co/papers/2104.07857) allows offloading model states to the CPU and/or NVMe to save even more memory. Smart partitioning and tiling algorithms allow each GPU to send and receive very small amounts of data during offloading such that a modern NVMe can fit an even larger total memory pool than is available to your training process. ZeRO-Infinity requires ZeRO-3.

Depending on the CPU and/or NVMe memory available, you can offload both the [optimizer states](https://www.deepspeed.ai/docs/config-json/#optimizer-offloading) and [parameters](https://www.deepspeed.ai/docs/config-json/#parameter-offloading), just one of them, or none. You should also make sure the `nvme_path` is pointing to an NVMe device, because while it still works with a normal hard drive or solid state drive, it'll be significantly slower. With a modern NVMe, you can expect peak transfer speeds of ~3.5GB/s for read and ~3GB/s for write operations. Lastly, [run a benchmark](https://github.com/microsoft/DeepSpeed/issues/998) on your training setup to determine the optimal `aio` configuration.

The example ZeRO-3/Infinity configuration file below sets most of the parameter values to `auto`, but you could also manually add these values.


key: _node_content
string_value:
{"id_": "d4b78d2f-9e69-4c89-bedc-f48627bd9d76", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b804e41c-c024-415e-a237-d023e8522938", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f5b32c4ee622bfdd02689dfb0fa3dd18dfd12c597d133bea3fce4630f197d719", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f4b0412c-2b3d-49fc-a9e8-7e757906fe88", "node_type": "1", "metadata": {}, "hash": "d0ad19e32536166afaaebb7c0efb03d77c88bb33534ba2ee2707a5f15ba7b325", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17186, "end_char_idx": 20541, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

</hfoption>
</hfoptions>

### NVMe configuration

[ZeRO-Infinity](https://hf.co/papers/2104.07857) allows offloading model states to the CPU and/or NVMe to save even more memory. Smart partitioning and tiling algorithms allow each GPU to send and receive very small amounts of data during offloading such that a modern NVMe can fit an even larger total memory pool than is available to your training process. ZeRO-Infinity requires ZeRO-3.

Depending on the CPU and/or NVMe memory available, you can offload both the [optimizer states](https://www.deepspeed.ai/docs/config-json/#optimizer-offloading) and [parameters](https://www.deepspeed.ai/docs/config-json/#parameter-offloading), just one of them, or none. You should also make sure the `nvme_path` is pointing to an NVMe device, because while it still works with a normal hard drive or solid state drive, it'll be significantly slower. With a modern NVMe, you can expect peak transfer speeds of ~3.5GB/s for read and ~3GB/s for write operations. Lastly, [run a benchmark](https://github.com/microsoft/DeepSpeed/issues/998) on your training setup to determine the optimal `aio` configuration.

The example ZeRO-3/Infinity configuration file below sets most of the parameter values to `auto`, but you could also manually add these values.

```yml
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },

    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto"
        }
    },

    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "nvme",
            "nvme_path": "/local_nvme",
            "pin_memory": true,
            "buffer_count": 4,
            "fast_init": false
        },
        "offload_param": {
            "device": "nvme",
            "nvme_path": "/local_nvme",
            "pin_memory": true,
            "buffer_count": 5,
            "buffer_size": 1e8,
            "max_in_cpu": 1e9
        },
        "aio": {
            "block_size": 262144,
            "queue_depth": 32,
            "thread_count": 1,
            "single_submit": false,
            "overlap_events": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_16bit_weights_on_model_save": true
    },

    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "steps_per_print": 2000,
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false
}
```

## DeepSpeed features

There are a number of important parameters to specify in the DeepSpeed configuration file which are briefly described in this section.


key: _node_content
string_value:
{"id_": "f4b0412c-2b3d-49fc-a9e8-7e757906fe88", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d4b78d2f-9e69-4c89-bedc-f48627bd9d76", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "db90bc862d291aaef1e368e2cb47f7860414a0f5a2eb5ea5683a3c8353f3061e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d74b81b4-022a-45e3-b16c-258fbcc75cda", "node_type": "1", "metadata": {}, "hash": "f61da2562c3fc580e05e27938d999e344cfb0e3768042539cf5b106e62452d0a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20543, "end_char_idx": 23457, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Activation/gradient checkpointing

Activation and gradient checkpointing trades speed for more GPU memory which allows you to overcome scenarios where your GPU is out of memory or to increase your batch size for better performance. To enable this feature:

1. For a Hugging Face model, set `model.gradient_checkpointing_enable()` or `--gradient_checkpointing` in the [`Trainer`].
2. For a non-Hugging Face model, use the DeepSpeed [Activation Checkpointing API](https://deepspeed.readthedocs.io/en/latest/activation-checkpointing.html). You could also replace the Transformers modeling code and replace `torch.utils.checkpoint` with the DeepSpeed API. This approach is more flexible because you can offload the forward activations to the CPU memory instead of recalculating them.

### Optimizer and scheduler

DeepSpeed and Transformers optimizer and scheduler can be mixed and matched as long as you don't enable `offload_optimizer`. When `offload_optimizer` is enabled, you could use a non-DeepSpeed optimizer (except for LAMB) as long as it has both a CPU and GPU implementation.

<Tip warning={true}>

The optimizer and scheduler parameters for the config file can be set from the command line to avoid hard to find errors. For example, if the learning rate is set to a different value in another place you can override it from the command line. Aside from the optimizer and scheduler parameters, you'll need to ensure your [`Trainer`] command line arguments match the DeepSpeed configuration.

</Tip>

<hfoptions id="opt-sched">
<hfoption id="optimizer">

DeepSpeed offers several [optimizers](https://www.deepspeed.ai/docs/config-json/#optimizer-parameters) (Adam, AdamW, OneBitAdam, and LAMB) but you can also import other optimizers from PyTorch. If you don't configure the optimizer in the config, the [`Trainer`] automatically selects AdamW and either uses the supplied values or the default values for the following parameters from the command line: `lr`, `adam_beta1`, `adam_beta2`, `adam_epsilon`, `weight_decay`.

You can set the parameters to `"auto"` or manually input your own desired values.

```yaml
{
   "optimizer": {
       "type": "AdamW",
       "params": {
         "lr": "auto",
         "betas": "auto",
         "eps": "auto",
         "weight_decay": "auto"
       }
   }
}
```

You can also use an unsupported optimizer by adding the following to the top level configuration.

```yaml
{
   "zero_allow_untested_optimizer": true
}
```

From DeepSpeed==0.8.3 on, if you want to use offload, you'll also need to the following to the top level configuration because offload works best with DeepSpeed's CPU Adam optimizer.

```yaml
{
   "zero_force_ds_cpu_optimizer": false
}
```

</hfoption>
<hfoption id="scheduler">

DeepSpeed supports the LRRangeTest, OneCycle, WarmupLR and WarmupDecayLR learning rate [schedulers](https://www.deepspeed.ai/docs/config-json/#scheduler-parameters).


key: _node_content
string_value:
{"id_": "d74b81b4-022a-45e3-b16c-258fbcc75cda", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f4b0412c-2b3d-49fc-a9e8-7e757906fe88", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "b559fbb0956f81a9a96432ddf855e22564d522b3fc2b005e0e1d53f8d49a6bf6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "df85f57c-4ac2-47e4-9cb2-18ce1ac6235a", "node_type": "1", "metadata": {}, "hash": "6157b46c7ac0d7a4fb4c6baf996d039d338e36630066789becda8eb83bd070b6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21897, "end_char_idx": 24877, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Aside from the optimizer and scheduler parameters, you'll need to ensure your [`Trainer`] command line arguments match the DeepSpeed configuration.

</Tip>

<hfoptions id="opt-sched">
<hfoption id="optimizer">

DeepSpeed offers several [optimizers](https://www.deepspeed.ai/docs/config-json/#optimizer-parameters) (Adam, AdamW, OneBitAdam, and LAMB) but you can also import other optimizers from PyTorch. If you don't configure the optimizer in the config, the [`Trainer`] automatically selects AdamW and either uses the supplied values or the default values for the following parameters from the command line: `lr`, `adam_beta1`, `adam_beta2`, `adam_epsilon`, `weight_decay`.

You can set the parameters to `"auto"` or manually input your own desired values.

```yaml
{
   "optimizer": {
       "type": "AdamW",
       "params": {
         "lr": "auto",
         "betas": "auto",
         "eps": "auto",
         "weight_decay": "auto"
       }
   }
}
```

You can also use an unsupported optimizer by adding the following to the top level configuration.

```yaml
{
   "zero_allow_untested_optimizer": true
}
```

From DeepSpeed==0.8.3 on, if you want to use offload, you'll also need to the following to the top level configuration because offload works best with DeepSpeed's CPU Adam optimizer.

```yaml
{
   "zero_force_ds_cpu_optimizer": false
}
```

</hfoption>
<hfoption id="scheduler">

DeepSpeed supports the LRRangeTest, OneCycle, WarmupLR and WarmupDecayLR learning rate [schedulers](https://www.deepspeed.ai/docs/config-json/#scheduler-parameters).

Transformers and DeepSpeed provide two of the same schedulers:

* WarmupLR is the same as `--lr_scheduler_type constant_with_warmup` in Transformers
* WarmupDecayLR is the same as  `--lr_scheduler_type linear` in Transformers (this is the default scheduler used in Transformers)

If you don't configure the scheduler in the config, the [`Trainer`] automatically selects WarmupDecayLR and either uses the supplied values or the default values for the following parameters from the command line: `warmup_min_lr`, `warmup_max_lr`, `warmup_num_steps`, `total_num_steps` (automatically calculated during run time if `max_steps` is not provided).

You can set the parameters to `"auto"` or manually input your own desired values.

```yaml
{
   "scheduler": {
         "type": "WarmupDecayLR",
         "params": {
             "total_num_steps": "auto",
             "warmup_min_lr": "auto",
             "warmup_max_lr": "auto",
             "warmup_num_steps": "auto"
         }
     }
}
```

</hfoption>
</hfoptions>

### Precision

Deepspeed supports fp32, fp16, and bf16 mixed precision.

<hfoptions id="precision">
<hfoption id="fp32">

If your model doesn't work well with mixed precision, for example if it wasn't pretrained in mixed precision, you may encounter overflow or underflow issues which can cause NaN loss. For these cases, you should use full fp32 precision by explicitly disabling the default fp16 mode.


key: _node_content
string_value:
{"id_": "df85f57c-4ac2-47e4-9cb2-18ce1ac6235a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d74b81b4-022a-45e3-b16c-258fbcc75cda", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "528a356e6774632f97776f9ceff43d3513e2a6593b7bf6725dd195e2e85a8c88", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6508cefa-bb21-4ae6-b36d-5158c5b6c562", "node_type": "1", "metadata": {}, "hash": "52a70a2b2daf4af82290a537120db619cd73c142ccf1b14b8aad6461254731c5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23459, "end_char_idx": 26459, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Transformers and DeepSpeed provide two of the same schedulers:

* WarmupLR is the same as `--lr_scheduler_type constant_with_warmup` in Transformers
* WarmupDecayLR is the same as  `--lr_scheduler_type linear` in Transformers (this is the default scheduler used in Transformers)

If you don't configure the scheduler in the config, the [`Trainer`] automatically selects WarmupDecayLR and either uses the supplied values or the default values for the following parameters from the command line: `warmup_min_lr`, `warmup_max_lr`, `warmup_num_steps`, `total_num_steps` (automatically calculated during run time if `max_steps` is not provided).

You can set the parameters to `"auto"` or manually input your own desired values.

```yaml
{
   "scheduler": {
         "type": "WarmupDecayLR",
         "params": {
             "total_num_steps": "auto",
             "warmup_min_lr": "auto",
             "warmup_max_lr": "auto",
             "warmup_num_steps": "auto"
         }
     }
}
```

</hfoption>
</hfoptions>

### Precision

Deepspeed supports fp32, fp16, and bf16 mixed precision.

<hfoptions id="precision">
<hfoption id="fp32">

If your model doesn't work well with mixed precision, for example if it wasn't pretrained in mixed precision, you may encounter overflow or underflow issues which can cause NaN loss. For these cases, you should use full fp32 precision by explicitly disabling the default fp16 mode.

```yaml
{
    "fp16": {
        "enabled": false
    }
}
```

For Ampere GPUs and PyTorch > 1.7, it automatically switches to the more efficient [tf32](https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices) format for some operations but the results are still in fp32. You can control it from the [`Trainer`] by setting `--tf32` to enable it, and `--tf32 0` or `--no_tf32` to disable it.

</hfoption>
<hfoption id="fp16">

To configure PyTorch AMP-like fp16 mixed precision reduces memory usage and accelerates training speed. [`Trainer`] automatically enables or disables fp16 based on the value of `args.fp16_backend`, and the rest of the config can be set by you. fp16 is enabled from the command line when the following arguments are passed: `--fp16`, `--fp16_backend amp` or `--fp16_full_eval`.

```yaml
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    }
}
```

For additional DeepSpeed fp16 training options, take a look at the [FP16 Training Options](https://www.deepspeed.ai/docs/config-json/#fp16-training-options) reference.

To configure Apex-like fp16 mixed precision, setup the config as shown below with `"auto"` or your own values. [`Trainer`] automatically configure `amp` based on the values of `args.fp16_backend` and `args.fp16_opt_level`. It can also be enabled from the command line when the following arguments are passed: `--fp16`, `--fp16_backend apex` or `--fp16_opt_level 01`.


key: _node_content
string_value:
{"id_": "6508cefa-bb21-4ae6-b36d-5158c5b6c562", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df85f57c-4ac2-47e4-9cb2-18ce1ac6235a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "1fb0aa5a6526058b32782a2685989b7a0a6b9c9d82bd15ee8a6fd2711466e177", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c432cd6-1c7a-4c95-b4af-487a3e47df03", "node_type": "1", "metadata": {}, "hash": "00d03e2729c3b10f3482d59bccd16eec56cbeece104fad2ed83a5d018a6821b5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25180, "end_char_idx": 28218, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can control it from the [`Trainer`] by setting `--tf32` to enable it, and `--tf32 0` or `--no_tf32` to disable it.

</hfoption>
<hfoption id="fp16">

To configure PyTorch AMP-like fp16 mixed precision reduces memory usage and accelerates training speed. [`Trainer`] automatically enables or disables fp16 based on the value of `args.fp16_backend`, and the rest of the config can be set by you. fp16 is enabled from the command line when the following arguments are passed: `--fp16`, `--fp16_backend amp` or `--fp16_full_eval`.

```yaml
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    }
}
```

For additional DeepSpeed fp16 training options, take a look at the [FP16 Training Options](https://www.deepspeed.ai/docs/config-json/#fp16-training-options) reference.

To configure Apex-like fp16 mixed precision, setup the config as shown below with `"auto"` or your own values. [`Trainer`] automatically configure `amp` based on the values of `args.fp16_backend` and `args.fp16_opt_level`. It can also be enabled from the command line when the following arguments are passed: `--fp16`, `--fp16_backend apex` or `--fp16_opt_level 01`.

```yaml
{
    "amp": {
        "enabled": "auto",
        "opt_level": "auto"
    }
}
```

</hfoption>
<hfoption id="bf16">

To use bf16, you'll need at least DeepSpeed==0.6.0. bf16 has the same dynamic range as fp32 and doesn’t require loss scaling. However, if you use [gradient accumulation](#gradient-accumulation) with bf16, gradients are accumulated in bf16 which may not be desired because this format's low precision can lead to lossy accumulation.

bf16 can be setup in the config file or enabled from the command line when the following arguments are passed: `--bf16` or `--bf16_full_eval`.

```yaml
{
    "bf16": {
        "enabled": "auto"
    }
}
```

</hfoption>
</hfoptions>

### Batch size

The batch size can be auto-configured or explicitly set. If you choose to use the `"auto"` option, [`Trainer`] sets `train_micro_batch_size_per_gpu` to the value of args.`per_device_train_batch_size` and `train_batch_size` to `args.world_size * args.per_device_train_batch_size * args.gradient_accumulation_steps`.

```yaml
{
    "train_micro_batch_size_per_gpu": "auto",
    "train_batch_size": "auto"
}
```

### Gradient accumulation

Gradient accumulation can be auto-configured or explicitly set. If you choose to use the `"auto"` option, [`Trainer`] sets it to the value of `args.gradient_accumulation_steps`.

```yaml
{
    "gradient_accumulation_steps": "auto"
}

```

### Gradient clipping

Gradient clipping can be auto-configured or explicitly set. If you choose to use the `"auto"` option, [`Trainer`] sets it to the value of `args.max_grad_norm`.

```yaml
{
    "gradient_clipping": "auto"
}
```

### Communication data type

For communication collectives like reduction, gathering and scattering operations, a separate data type is used.


key: _node_content
string_value:
{"id_": "0c432cd6-1c7a-4c95-b4af-487a3e47df03", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6508cefa-bb21-4ae6-b36d-5158c5b6c562", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "8bea0ff72130c30c82329843101e053dd3e7c67f0a475a30431f668b733cc4e3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ca9aa9a-a5e4-45ac-87ee-17d695264654", "node_type": "1", "metadata": {}, "hash": "ee5ca824608638ebfc9b08113de83051a2fe34d9c4b6ef203cd30619184d2f5c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26638, "end_char_idx": 29889, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
bf16 has the same dynamic range as fp32 and doesn’t require loss scaling. However, if you use [gradient accumulation](#gradient-accumulation) with bf16, gradients are accumulated in bf16 which may not be desired because this format's low precision can lead to lossy accumulation.

bf16 can be setup in the config file or enabled from the command line when the following arguments are passed: `--bf16` or `--bf16_full_eval`.

```yaml
{
    "bf16": {
        "enabled": "auto"
    }
}
```

</hfoption>
</hfoptions>

### Batch size

The batch size can be auto-configured or explicitly set. If you choose to use the `"auto"` option, [`Trainer`] sets `train_micro_batch_size_per_gpu` to the value of args.`per_device_train_batch_size` and `train_batch_size` to `args.world_size * args.per_device_train_batch_size * args.gradient_accumulation_steps`.

```yaml
{
    "train_micro_batch_size_per_gpu": "auto",
    "train_batch_size": "auto"
}
```

### Gradient accumulation

Gradient accumulation can be auto-configured or explicitly set. If you choose to use the `"auto"` option, [`Trainer`] sets it to the value of `args.gradient_accumulation_steps`.

```yaml
{
    "gradient_accumulation_steps": "auto"
}

```

### Gradient clipping

Gradient clipping can be auto-configured or explicitly set. If you choose to use the `"auto"` option, [`Trainer`] sets it to the value of `args.max_grad_norm`.

```yaml
{
    "gradient_clipping": "auto"
}
```

### Communication data type

For communication collectives like reduction, gathering and scattering operations, a separate data type is used.

All gather and scatter operations are performed in the same data type the data is in. For example, if you're training with bf16, the data is also gathered in bf16 because gathering is a non-lossy operation.

Reduce operations are lossy, for example when gradients are averaged across multiple GPUs. When the communication is done in fp16 or bf16, it is more likely to be lossy because adding multiple numbers in low precision isn't exact. This is especially the case with bf16 which has a lower precision than fp16. For this reason, fp16 is the default for reduction operations because the loss is minimal when averaging gradients.

You can choose the communication data type by setting the `communication_data_type` parameter in the config file. For example, choosing fp32 adds a small amount of overhead but ensures the reduction operation is accumulated in fp32 and when it is ready, it is downcasted to whichever half-precision dtype you're training in.

```yaml
{
    "communication_data_type": "fp32"
}
```

## Deployment

DeepSpeed can be deployed by different launchers such as [torchrun](https://pytorch.org/docs/stable/elastic/run.html), the `deepspeed` launcher, or [Accelerate](https://huggingface.co/docs/accelerate/basic_tutorials/launch#using-accelerate-launch). To deploy, add `--deepspeed ds_config.json` to the [`Trainer`] command line. It’s recommended to use DeepSpeed’s [`add_config_arguments`](https://deepspeed.readthedocs.io/en/latest/initialize.html#argument-parsing) utility to add any necessary command line arguments to your code.

This guide will show you how to deploy DeepSpeed with the `deepspeed` launcher for different training setups.


key: _node_content
string_value:
{"id_": "5ca9aa9a-a5e4-45ac-87ee-17d695264654", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c432cd6-1c7a-4c95-b4af-487a3e47df03", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "6f47fa40ac9be48b8a39348cde637bdbb02bd06b21b3e2eb161d6698769a589c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "63d11a03-12b4-45f2-a6ba-21234995f7c5", "node_type": "1", "metadata": {}, "hash": "b5ddc7f78eac095e2e047f5704896e9394510728b35efb14e9141b850a54fe8d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28220, "end_char_idx": 31385, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All gather and scatter operations are performed in the same data type the data is in. For example, if you're training with bf16, the data is also gathered in bf16 because gathering is a non-lossy operation.

Reduce operations are lossy, for example when gradients are averaged across multiple GPUs. When the communication is done in fp16 or bf16, it is more likely to be lossy because adding multiple numbers in low precision isn't exact. This is especially the case with bf16 which has a lower precision than fp16. For this reason, fp16 is the default for reduction operations because the loss is minimal when averaging gradients.

You can choose the communication data type by setting the `communication_data_type` parameter in the config file. For example, choosing fp32 adds a small amount of overhead but ensures the reduction operation is accumulated in fp32 and when it is ready, it is downcasted to whichever half-precision dtype you're training in.

```yaml
{
    "communication_data_type": "fp32"
}
```

## Deployment

DeepSpeed can be deployed by different launchers such as [torchrun](https://pytorch.org/docs/stable/elastic/run.html), the `deepspeed` launcher, or [Accelerate](https://huggingface.co/docs/accelerate/basic_tutorials/launch#using-accelerate-launch). To deploy, add `--deepspeed ds_config.json` to the [`Trainer`] command line. It’s recommended to use DeepSpeed’s [`add_config_arguments`](https://deepspeed.readthedocs.io/en/latest/initialize.html#argument-parsing) utility to add any necessary command line arguments to your code.

This guide will show you how to deploy DeepSpeed with the `deepspeed` launcher for different training setups. You can check out this [post](https://github.com/huggingface/transformers/issues/8771#issuecomment-759248400) for more practical usage examples.


<hfoptions id="deploy">
<hfoption id="multi-GPU">

To deploy DeepSpeed on multiple GPUs, add the `--num_gpus` parameter. If you want to use all available GPUs, you don't need to add `--num_gpus`. The example below uses 2 GPUs.

```bash
deepspeed --num_gpus=2 examples/pytorch/translation/run_translation.py \
--deepspeed tests/deepspeed/ds_config_zero3.json \
--model_name_or_path google-t5/t5-small --per_device_train_batch_size 1 \
--output_dir output_dir --overwrite_output_dir --fp16 \
--do_train --max_train_samples 500 --num_train_epochs 1 \
--dataset_name wmt16 --dataset_config "ro-en" \
--source_lang en --target_lang ro
```

</hfoption>
<hfoption id="single-GPU">

To deploy DeepSpeed on a single GPU, add the `--num_gpus` parameter. It isn't necessary to explicitly set this value if you only have 1 GPU because DeepSpeed deploys all GPUs it can see on a given node.

```bash
deepspeed --num_gpus=1 examples/pytorch/translation/run_translation.py \
--deepspeed tests/deepspeed/ds_config_zero2.json \
--model_name_or_path google-t5/t5-small --per_device_train_batch_size 1 \
--output_dir output_dir --overwrite_output_dir --fp16 \
--do_train --max_train_samples 500 --num_train_epochs 1 \
--dataset_name wmt16 --dataset_config "ro-en" \
--source_lang en --target_lang ro
```

DeepSpeed is still useful with just 1 GPU because you can:

1.


key: _node_content
string_value:
{"id_": "63d11a03-12b4-45f2-a6ba-21234995f7c5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ca9aa9a-a5e4-45ac-87ee-17d695264654", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "ee7c2045ad8bcbcb3f505ffbfab8c50d85021fd8ba1316493bf2e37b3ccad00e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6b5b2c19-cd56-484e-bed1-0e78c4265904", "node_type": "1", "metadata": {}, "hash": "8400c1488b6c431fafc961f08640f129123276ea18f01ef5fe314bd62733e3c1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30037, "end_char_idx": 33159, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<hfoptions id="deploy">
<hfoption id="multi-GPU">

To deploy DeepSpeed on multiple GPUs, add the `--num_gpus` parameter. If you want to use all available GPUs, you don't need to add `--num_gpus`. The example below uses 2 GPUs.

```bash
deepspeed --num_gpus=2 examples/pytorch/translation/run_translation.py \
--deepspeed tests/deepspeed/ds_config_zero3.json \
--model_name_or_path google-t5/t5-small --per_device_train_batch_size 1 \
--output_dir output_dir --overwrite_output_dir --fp16 \
--do_train --max_train_samples 500 --num_train_epochs 1 \
--dataset_name wmt16 --dataset_config "ro-en" \
--source_lang en --target_lang ro
```

</hfoption>
<hfoption id="single-GPU">

To deploy DeepSpeed on a single GPU, add the `--num_gpus` parameter. It isn't necessary to explicitly set this value if you only have 1 GPU because DeepSpeed deploys all GPUs it can see on a given node.

```bash
deepspeed --num_gpus=1 examples/pytorch/translation/run_translation.py \
--deepspeed tests/deepspeed/ds_config_zero2.json \
--model_name_or_path google-t5/t5-small --per_device_train_batch_size 1 \
--output_dir output_dir --overwrite_output_dir --fp16 \
--do_train --max_train_samples 500 --num_train_epochs 1 \
--dataset_name wmt16 --dataset_config "ro-en" \
--source_lang en --target_lang ro
```

DeepSpeed is still useful with just 1 GPU because you can:

1. Offload some computations and memory to the CPU to make more GPU resources available to your model to use a larger batch size or fit a very large model that normally won't fit.
2. Minimize memory fragmentation with it's smart GPU memory management system which also allows you to fit bigger models and data batches.

<Tip>

Set the `allgather_bucket_size` and `reduce_bucket_size` values to 2e8 in the [ZeRO-2](#zero-configuration) configuration file to get better performance on a single GPU.

</Tip>

</hfoption>
</hfoptions>

### Multi-node deployment

A node is one or more GPUs for running a workload. A more powerful setup is a multi-node setup which can be launched with the `deepspeed` launcher. For this guide, let's assume there are two nodes with 8 GPUs each. The first node can be accessed `ssh hostname1` and the second node with `ssh hostname2`. Both nodes must be able to communicate with each other locally over ssh without a password.

By default, DeepSpeed expects your multi-node environment to use a shared storage. If this is not the case and each node can only see the local filesystem, you need to adjust the config file to include a [`checkpoint`](https://www.deepspeed.ai/docs/config-json/#checkpoint-options) to allow loading without access to a shared filesystem:

```yaml
{
  "checkpoint": {
    "use_node_local_storage": true
  }
}
```

You could also use the [`Trainer`]'s `--save_on_each_node` argument to automatically add the above `checkpoint` to your config.

<hfoptions id="multinode">
<hfoption id="torchrun">

For [torchrun](https://pytorch.org/docs/stable/elastic/run.html), you have to ssh to each node and run the following command on both of them. The launcher waits until both nodes are synchronized before launching the training.


key: _node_content
string_value:
{"id_": "6b5b2c19-cd56-484e-bed1-0e78c4265904", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "63d11a03-12b4-45f2-a6ba-21234995f7c5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "45285b8914e837cc47e5510c45fd84e60a3f99485c03fe9f266f4461b5f4a018", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a64c6cd6-2ccc-4247-b9dc-9f4adf42f59c", "node_type": "1", "metadata": {}, "hash": "6700ab39d5b5a7bf4cb4580fc20616a9a663fb33e6871c17725f4588b307e8d8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31563, "end_char_idx": 34355, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


