# embedding_metadata

key: chroma:document
string_value:
In a few lines of code, you can harness the power of an LLM.


## Common pitfalls

There are many [generation strategies](generation_strategies), and sometimes the default values may not be appropriate for your use case. If your outputs aren't aligned with what you're expecting, we've created a list of the most common pitfalls and how to avoid them.

```py
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
... )
```

### Generated output is too short/long

If not specified in the [`~generation.GenerationConfig`] file, `generate` returns up to 20 tokens by default. We highly recommend manually setting `max_new_tokens` in your `generate` call to control the maximum number of new tokens it can return. Keep in mind LLMs (more precisely, [decoder-only models](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt)) also return the input prompt as part of the output.


```py
>>> model_inputs = tokenizer(["A sequence of numbers: 1, 2"], return_tensors="pt").to("cuda")

>>> # By default, the output will contain up to 20 tokens
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A sequence of numbers: 1, 2, 3, 4, 5'

>>> # Setting `max_new_tokens` allows you to control the maximum length
>>> generated_ids = model.generate(**model_inputs, max_new_tokens=50)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A sequence of numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,'
```

### Incorrect generation mode

By default, and unless specified in the [`~generation.GenerationConfig`] file, `generate` selects the most likely token at each iteration (greedy decoding). Depending on your task, this may be undesirable; creative tasks like chatbots or writing an essay benefit from sampling. On the other hand, input-grounded tasks like audio transcription or translation benefit from greedy decoding. Enable sampling with `do_sample=True`, and you can learn more about this topic in this [blog post](https://huggingface.co/blog/how-to-generate).

```py
>>> # Set seed or reproducibility -- you don't need this unless you want full reproducibility
>>> from transformers import set_seed
>>> set_seed(42)

>>> model_inputs = tokenizer(["I am a cat."], return_tensors="pt").to("cuda")

>>> # LLM + greedy decoding = repetitive, boring output
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat. I am a cat. I am a cat. I am a cat'

>>> # With sampling, the output becomes more creative!
>>> generated_ids = model.generate(**model_inputs, do_sample=True)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat.  Specifically, I am an indoor-only cat.


key: _node_content
string_value:
{"id_": "becdd942-28e6-45cf-85e0-84caf4a7d226", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4dc989a4-6a0c-4365-9654-5612eb0302da", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "82fec938d93e159032195b730593f8173f2a6ec5cbffcabb38446be519a053d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ebbaf416-74c3-45f8-ac85-d93b4eb36c12", "node_type": "1", "metadata": {}, "hash": "c4ae41c5777a36ba310ec9edb457541ab67e3b5379e0b262c94d74b7cd631834", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7942, "end_char_idx": 10796, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Depending on your task, this may be undesirable; creative tasks like chatbots or writing an essay benefit from sampling. On the other hand, input-grounded tasks like audio transcription or translation benefit from greedy decoding. Enable sampling with `do_sample=True`, and you can learn more about this topic in this [blog post](https://huggingface.co/blog/how-to-generate).

```py
>>> # Set seed or reproducibility -- you don't need this unless you want full reproducibility
>>> from transformers import set_seed
>>> set_seed(42)

>>> model_inputs = tokenizer(["I am a cat."], return_tensors="pt").to("cuda")

>>> # LLM + greedy decoding = repetitive, boring output
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat. I am a cat. I am a cat. I am a cat'

>>> # With sampling, the output becomes more creative!
>>> generated_ids = model.generate(**model_inputs, do_sample=True)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat.  Specifically, I am an indoor-only cat.  I'
```

### Wrong padding side

LLMs are [decoder-only](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt) architectures, meaning they continue to iterate on your input prompt. If your inputs do not have the same length, they need to be padded. Since LLMs are not trained to continue from pad tokens, your input needs to be left-padded. Make sure you also don't forget to pass the attention mask to generate!

```py
>>> # The tokenizer initialized above has right-padding active by default: the 1st sequence,
>>> # which is shorter, has padding on the right side. Generation fails to capture the logic.
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'1, 2, 33333333333'

>>> # With left-padding, it works as expected!
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'1, 2, 3, 4, 5, 6,'
```

### Wrong prompt

Some models and tasks expect a certain input prompt format to work properly. When this format is not applied, you will get a silent performance degradation: the model kinda works, but not as well as if you were following the expected prompt. More information about prompting, including which models and tasks need to be careful, is available in this [guide](tasks/prompting).


key: _node_content
string_value:
{"id_": "ebbaf416-74c3-45f8-ac85-d93b4eb36c12", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "becdd942-28e6-45cf-85e0-84caf4a7d226", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "662e52ce184091124a19c57bcd9737da77da0602ce0491405c5f2054026fb796", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a2782a97-7778-4d3b-90b5-30d3f7c8cffc", "node_type": "1", "metadata": {}, "hash": "a89520078fde28d8d36aadfd5688f9bed45706a5660444e89f25497e5b21f980", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9281, "end_char_idx": 12048, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since LLMs are not trained to continue from pad tokens, your input needs to be left-padded. Make sure you also don't forget to pass the attention mask to generate!

```py
>>> # The tokenizer initialized above has right-padding active by default: the 1st sequence,
>>> # which is shorter, has padding on the right side. Generation fails to capture the logic.
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'1, 2, 33333333333'

>>> # With left-padding, it works as expected!
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'1, 2, 3, 4, 5, 6,'
```

### Wrong prompt

Some models and tasks expect a certain input prompt format to work properly. When this format is not applied, you will get a silent performance degradation: the model kinda works, but not as well as if you were following the expected prompt. More information about prompting, including which models and tasks need to be careful, is available in this [guide](tasks/prompting). Let's see an example with a chat LLM, which makes use of [chat templating](chat_templating):

```python
>>> tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-alpha")
>>> model = AutoModelForCausalLM.from_pretrained(
...     "HuggingFaceH4/zephyr-7b-alpha", device_map="auto", load_in_4bit=True
... )
>>> set_seed(0)
>>> prompt = """How many helicopters can a human eat in one sitting? Reply as a thug."""
>>> model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
>>> input_length = model_inputs.input_ids.shape[1]
>>> generated_ids = model.generate(**model_inputs, max_new_tokens=20)
>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])
"I'm not a thug, but i can tell you that a human cannot eat"
>>> # Oh no, it did not follow our instruction to reply as a thug! Let's see what happens when we write
>>> # a better prompt and use the right template for this model (through `tokenizer.apply_chat_template`)

>>> set_seed(0)
>>> messages = [
...     {
...         "role": "system",
...         "content": "You are a friendly chatbot who always responds in the style of a thug",
...     },
...     {"role": "user", "content": "How many helicopters can a human eat in one sitting?


key: _node_content
string_value:
{"id_": "a2782a97-7778-4d3b-90b5-30d3f7c8cffc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ebbaf416-74c3-45f8-ac85-d93b4eb36c12", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4ae5fc98018d7205edf19ce69f3b6c5ce4382a2b1f1b98e159549412b5cbd064", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8832e1e1-0c42-4871-9216-90d0b19f0b5b", "node_type": "1", "metadata": {}, "hash": "10a427314a0a5650b9c27bd9a76b199f2b2314d69d4b7604ecbef6aab780a3d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10498, "end_char_idx": 13715, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When this format is not applied, you will get a silent performance degradation: the model kinda works, but not as well as if you were following the expected prompt. More information about prompting, including which models and tasks need to be careful, is available in this [guide](tasks/prompting). Let's see an example with a chat LLM, which makes use of [chat templating](chat_templating):

```python
>>> tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-alpha")
>>> model = AutoModelForCausalLM.from_pretrained(
...     "HuggingFaceH4/zephyr-7b-alpha", device_map="auto", load_in_4bit=True
... )
>>> set_seed(0)
>>> prompt = """How many helicopters can a human eat in one sitting? Reply as a thug."""
>>> model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
>>> input_length = model_inputs.input_ids.shape[1]
>>> generated_ids = model.generate(**model_inputs, max_new_tokens=20)
>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])
"I'm not a thug, but i can tell you that a human cannot eat"
>>> # Oh no, it did not follow our instruction to reply as a thug! Let's see what happens when we write
>>> # a better prompt and use the right template for this model (through `tokenizer.apply_chat_template`)

>>> set_seed(0)
>>> messages = [
...     {
...         "role": "system",
...         "content": "You are a friendly chatbot who always responds in the style of a thug",
...     },
...     {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
... ]
>>> model_inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
>>> input_length = model_inputs.shape[1]
>>> generated_ids = model.generate(model_inputs, do_sample=True, max_new_tokens=20)
>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])
'None, you thug. How bout you try to focus on more useful questions?'
>>> # As we can see, it followed a proper thug style 😎
```

## Further resources

While the autoregressive generation process is relatively straightforward, making the most out of your LLM can be a challenging endeavor because there are many moving parts. For your next steps to help you dive deeper into LLM usage and understanding:

### Advanced generate usage

1. Guide on how to [control different generation methods](generation_strategies), how to set up the generation configuration file, and how to stream the output;
2. [Accelerating text generation](llm_optims);
3. [Prompt templates for chat LLMs](chat_templating);
4. [Prompt design guide](tasks/prompting);
5. API reference on [`~generation.GenerationConfig`], [`~generation.GenerationMixin.generate`], and [generate-related classes](internal/generation_utils). Most of the classes, including the logits processors, have usage examples!

### LLM leaderboards

1. [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), which focuses on the quality of the open-source models;
2. [Open LLM-Perf Leaderboard](https://huggingface.co/spaces/optimum/llm-perf-leaderboard), which focuses on LLM throughput.

### Latency, throughput and memory utilization

1.


key: _node_content
string_value:
{"id_": "8832e1e1-0c42-4871-9216-90d0b19f0b5b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a2782a97-7778-4d3b-90b5-30d3f7c8cffc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "be787d5e68952bc6194edc5656ae85b58daac23b2a32f91a230b48d24046b353", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12048, "end_char_idx": 14486, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
... ]
>>> model_inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
>>> input_length = model_inputs.shape[1]
>>> generated_ids = model.generate(model_inputs, do_sample=True, max_new_tokens=20)
>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])
'None, you thug. How bout you try to focus on more useful questions?'
>>> # As we can see, it followed a proper thug style 😎
```

## Further resources

While the autoregressive generation process is relatively straightforward, making the most out of your LLM can be a challenging endeavor because there are many moving parts. For your next steps to help you dive deeper into LLM usage and understanding:

### Advanced generate usage

1. Guide on how to [control different generation methods](generation_strategies), how to set up the generation configuration file, and how to stream the output;
2. [Accelerating text generation](llm_optims);
3. [Prompt templates for chat LLMs](chat_templating);
4. [Prompt design guide](tasks/prompting);
5. API reference on [`~generation.GenerationConfig`], [`~generation.GenerationMixin.generate`], and [generate-related classes](internal/generation_utils). Most of the classes, including the logits processors, have usage examples!

### LLM leaderboards

1. [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), which focuses on the quality of the open-source models;
2. [Open LLM-Perf Leaderboard](https://huggingface.co/spaces/optimum/llm-perf-leaderboard), which focuses on LLM throughput.

### Latency, throughput and memory utilization

1. Guide on how to [optimize LLMs for speed and memory](llm_tutorial_optimization);
2. Guide on [quantization](main_classes/quantization) such as bitsandbytes and autogptq, which shows you how to drastically reduce your memory requirements.

### Related libraries

1. [`optimum`](https://github.com/huggingface/optimum), an extension of 🤗 Transformers that optimizes for specific hardware devices.
2. [`outlines`](https://github.com/outlines-dev/outlines), a library where you can constrain text generation (e.g. to generate JSON files);
3. [`text-generation-inference`](https://github.com/huggingface/text-generation-inference), a production-ready server for LLMs;
4. [`text-generation-webui`](https://github.com/oobabooga/text-generation-webui), a UI for text generation;


key: _node_content
string_value:
{"id_": "4027a52e-5e56-4697-8b8e-b6540805578d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dbe7f2f375c2112c34c1b7ce0a45743df1c5bc33d4786c28f846ede83925cbcc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6a9542aa-ec25-4bbf-98b3-562e087e1007", "node_type": "1", "metadata": {}, "hash": "52e5ba33aed87ce49846f926e41404dc4fc7d133f0b6f5e5ad56f603cf8d21ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3273, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Export to TorchScript

<Tip>

This is the very beginning of our experiments with TorchScript and we are still
exploring its capabilities with variable-input-size models. It is a focus of interest to
us and we will deepen our analysis in upcoming releases, with more code examples, a more
flexible implementation, and benchmarks comparing Python-based codes with compiled
TorchScript.

</Tip>

According to the [TorchScript documentation](https://pytorch.org/docs/stable/jit.html):

> TorchScript is a way to create serializable and optimizable models from PyTorch code.

There are two PyTorch modules, [JIT and
TRACE](https://pytorch.org/docs/stable/jit.html), that allow developers to export their
models to be reused in other programs like efficiency-oriented C++ programs.

We provide an interface that allows you to export 🤗 Transformers models to TorchScript
so they can be reused in a different environment than PyTorch-based Python programs.
Here, we explain how to export and use our models using TorchScript.

Exporting a model requires two things:

- model instantiation with the `torchscript` flag
- a forward pass with dummy inputs

These necessities imply several things developers should be careful about as detailed
below.

## TorchScript flag and tied weights

The `torchscript` flag is necessary because most of the 🤗 Transformers language models
have tied weights between their `Embedding` layer and their `Decoding` layer.
TorchScript does not allow you to export models that have tied weights, so it is
necessary to untie and clone the weights beforehand.

Models instantiated with the `torchscript` flag have their `Embedding` layer and
`Decoding` layer separated, which means that they should not be trained down the line.
Training would desynchronize the two layers, leading to unexpected results.

This is not the case for models that do not have a language model head, as those do not
have tied weights. These models can be safely exported without the `torchscript` flag.

## Dummy inputs and standard lengths

The dummy inputs are used for a models forward pass. While the inputs' values are
propagated through the layers, PyTorch keeps track of the different operations executed
on each tensor. These recorded operations are then used to create the *trace* of the
model.

The trace is created relative to the inputs' dimensions. It is therefore constrained by
the dimensions of the dummy input, and will not work for any other sequence length or
batch size. When trying with a different size, the following error is raised:

```
`The expanded size of the tensor (3) must match the existing size (7) at non-singleton dimension 2`
```

We recommended you trace the model with a dummy input size at least as large as the
largest input that will be fed to the model during inference. Padding can help fill the
missing values. However, since the model is traced with a larger input size, the
dimensions of the matrix will also be large, resulting in more calculations.

Be careful of the total number of operations done on each input and follow the
performance closely when exporting varying sequence-length models.

## Using TorchScript in Python

This section demonstrates how to save and load models as well as how to use the trace
for inference.


key: _node_content
string_value:
{"id_": "6a9542aa-ec25-4bbf-98b3-562e087e1007", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dbe7f2f375c2112c34c1b7ce0a45743df1c5bc33d4786c28f846ede83925cbcc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4027a52e-5e56-4697-8b8e-b6540805578d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "79e20a55b702951e1fded9c7c149e4ce8070feea5919efbad7655a5b970a99c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bcb660d9-fb70-4fd7-8d51-47199ef7fbd1", "node_type": "1", "metadata": {}, "hash": "3e01464f563ce2f71b32676684c7482e3face88131c2997f8c07b500a5c2834d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1444, "end_char_idx": 4263, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
TorchScript does not allow you to export models that have tied weights, so it is
necessary to untie and clone the weights beforehand.

Models instantiated with the `torchscript` flag have their `Embedding` layer and
`Decoding` layer separated, which means that they should not be trained down the line.
Training would desynchronize the two layers, leading to unexpected results.

This is not the case for models that do not have a language model head, as those do not
have tied weights. These models can be safely exported without the `torchscript` flag.

## Dummy inputs and standard lengths

The dummy inputs are used for a models forward pass. While the inputs' values are
propagated through the layers, PyTorch keeps track of the different operations executed
on each tensor. These recorded operations are then used to create the *trace* of the
model.

The trace is created relative to the inputs' dimensions. It is therefore constrained by
the dimensions of the dummy input, and will not work for any other sequence length or
batch size. When trying with a different size, the following error is raised:

```
`The expanded size of the tensor (3) must match the existing size (7) at non-singleton dimension 2`
```

We recommended you trace the model with a dummy input size at least as large as the
largest input that will be fed to the model during inference. Padding can help fill the
missing values. However, since the model is traced with a larger input size, the
dimensions of the matrix will also be large, resulting in more calculations.

Be careful of the total number of operations done on each input and follow the
performance closely when exporting varying sequence-length models.

## Using TorchScript in Python

This section demonstrates how to save and load models as well as how to use the trace
for inference.

### Saving a model

To export a `BertModel` with TorchScript, instantiate `BertModel` from the `BertConfig`
class and then save it to disk under the filename `traced_bert.pt`:

```python
from transformers import BertModel, BertTokenizer, BertConfig
import torch

enc = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")

# Tokenizing input text
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)

# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = "[MASK]"
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

# Creating a dummy input
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]

# Initializing the model with the torchscript flag
# Flag set to True even though it is not necessary as this model does not have an LM Head.


key: _node_content
string_value:
{"id_": "bcb660d9-fb70-4fd7-8d51-47199ef7fbd1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dbe7f2f375c2112c34c1b7ce0a45743df1c5bc33d4786c28f846ede83925cbcc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6a9542aa-ec25-4bbf-98b3-562e087e1007", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "86eefb45f86937df42d55a68673b2c2051eae8379077366f56bd3a186475e8af", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c9dc1fdb-7b99-45d9-a7b6-2be16e38e778", "node_type": "1", "metadata": {}, "hash": "903ed2d58ac048bf9b1a1582a8992dc03fa1b128394875b48eb5f15d182779d9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2809, "end_char_idx": 5978, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Padding can help fill the
missing values. However, since the model is traced with a larger input size, the
dimensions of the matrix will also be large, resulting in more calculations.

Be careful of the total number of operations done on each input and follow the
performance closely when exporting varying sequence-length models.

## Using TorchScript in Python

This section demonstrates how to save and load models as well as how to use the trace
for inference.

### Saving a model

To export a `BertModel` with TorchScript, instantiate `BertModel` from the `BertConfig`
class and then save it to disk under the filename `traced_bert.pt`:

```python
from transformers import BertModel, BertTokenizer, BertConfig
import torch

enc = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")

# Tokenizing input text
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)

# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = "[MASK]"
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

# Creating a dummy input
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]

# Initializing the model with the torchscript flag
# Flag set to True even though it is not necessary as this model does not have an LM Head.
config = BertConfig(
    vocab_size_or_config_json_file=32000,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    torchscript=True,
)

# Instantiating the model
model = BertModel(config)

# The model needs to be in evaluation mode
model.eval()

# If you are instantiating the model with *from_pretrained* you can also easily set the TorchScript flag
model = BertModel.from_pretrained("google-bert/bert-base-uncased", torchscript=True)

# Creating the trace
traced_model = torch.jit.trace(model, [tokens_tensor, segments_tensors])
torch.jit.save(traced_model, "traced_bert.pt")
```

### Loading a model

Now you can load the previously saved `BertModel`, `traced_bert.pt`, from disk and use
it on the previously initialised `dummy_input`:

```python
loaded_model = torch.jit.load("traced_bert.pt")
loaded_model.eval()

all_encoder_layers, pooled_output = loaded_model(*dummy_input)
```

### Using a traced model for inference

Use the traced model for inference by using its `__call__` dunder method:

```python
traced_model(tokens_tensor, segments_tensors)
```

## Deploy Hugging Face TorchScript models to AWS with the Neuron SDK

AWS introduced the [Amazon EC2 Inf1](https://aws.amazon.com/ec2/instance-types/inf1/)
instance family for low cost, high performance machine learning inference in the cloud.
The Inf1 instances are powered by the AWS Inferentia chip, a custom-built hardware
accelerator, specializing in deep learning inferencing workloads. [AWS
Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/#) is the SDK for
Inferentia that supports tracing and optimizing transformers models for deployment on
Inf1. The Neuron SDK provides:


key: _node_content
string_value:
{"id_": "c9dc1fdb-7b99-45d9-a7b6-2be16e38e778", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dbe7f2f375c2112c34c1b7ce0a45743df1c5bc33d4786c28f846ede83925cbcc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bcb660d9-fb70-4fd7-8d51-47199ef7fbd1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8a3fc84f4d5a505e5a7d36e017b485488b19b4500b2f3299ac38151774351aee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5626, "end_char_idx": 8863, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Inf1 instances are powered by the AWS Inferentia chip, a custom-built hardware
accelerator, specializing in deep learning inferencing workloads. [AWS
Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/#) is the SDK for
Inferentia that supports tracing and optimizing transformers models for deployment on
Inf1. The Neuron SDK provides:


1. Easy-to-use API with one line of code change to trace and optimize a TorchScript
   model for inference in the cloud.
2. Out of the box performance optimizations for [improved
   cost-performance](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/benchmark/>).
3. Support for Hugging Face transformers models built with either
   [PyTorch](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/bert_tutorial/tutorial_pretrained_bert.html)
   or
   [TensorFlow](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/tensorflow/huggingface_bert/huggingface_bert.html).

### Implications

Transformers models based on the [BERT (Bidirectional Encoder Representations from
Transformers)](https://huggingface.co/docs/transformers/main/model_doc/bert)
architecture, or its variants such as
[distilBERT](https://huggingface.co/docs/transformers/main/model_doc/distilbert) and
[roBERTa](https://huggingface.co/docs/transformers/main/model_doc/roberta) run best on
Inf1 for non-generative tasks such as extractive question answering, sequence
classification, and token classification. However, text generation tasks can still be
adapted to run on Inf1 according to this [AWS Neuron MarianMT
tutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/transformers-marianmt.html).
More information about models that can be converted out of the box on Inferentia can be
found in the [Model Architecture
Fit](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/models/models-inferentia.html#models-inferentia)
section of the Neuron documentation.

### Dependencies

Using AWS Neuron to convert models requires a [Neuron SDK
environment](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-frameworks/pytorch-neuron/index.html#installation-guide)
which comes preconfigured on [AWS Deep Learning
AMI](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-inferentia-launching.html).

### Converting a model for AWS Neuron

Convert a model for AWS NEURON using the same code from [Using TorchScript in
Python](torchscript#using-torchscript-in-python) to trace a `BertModel`. Import the
`torch.neuron` framework extension to access the components of the Neuron SDK through a
Python API:

```python
from transformers import BertModel, BertTokenizer, BertConfig
import torch
import torch.neuron
```

You only need to modify the following line:

```diff
- torch.jit.trace(model, [tokens_tensor, segments_tensors])
+ torch.neuron.trace(model, [token_tensor, segments_tensors])
```

This enables the Neuron SDK to trace the model and optimize it for Inf1 instances.

To learn more about AWS Neuron SDK features, tools, example tutorials and latest
updates, please see the [AWS NeuronSDK
documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/index.html).


key: _node_content
string_value:
{"id_": "2a137cf0-b5ba-4c86-a95b-56942746cfb7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fde06c30-8b87-4a2e-a6d6-d1dbc82752c2", "node_type": "1", "metadata": {}, "hash": "d2d5e0eddc9d1b1deab50d5955d2a86fa61a989c75273fe3b9038b327fa732ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3323, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Training on TPU with TensorFlow

<Tip>

If you don't need long explanations and just want TPU code samples to get started with, check out [our TPU example notebook!](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb)

</Tip>

### What is a TPU?

A TPU is a **Tensor Processing Unit.** They are hardware designed by Google, which are used to greatly speed up the tensor computations within neural networks, much like GPUs. They can be used for both network training and inference. They are generally accessed through Google’s cloud services, but small TPUs can also be accessed directly for free through Google Colab and Kaggle Kernels.

Because [all TensorFlow models in 🤗 Transformers are Keras models](https://huggingface.co/blog/tensorflow-philosophy), most of the methods in this document are generally applicable to TPU training for any Keras model! However, there are a few points that are specific to the HuggingFace ecosystem (hug-o-system?) of Transformers and Datasets, and we’ll make sure to flag them up when we get to them.

### What kinds of TPU are available?

New users are often very confused by the range of TPUs, and the different ways to access them. The first key distinction to understand is the difference between **TPU Nodes** and **TPU VMs.**

When you use a **TPU Node**, you are effectively indirectly accessing a remote TPU. You will need a separate VM, which will initialize your network and data pipeline and then forward them to the remote node. When you use a TPU on Google Colab, you are accessing it in the **TPU Node** style.

Using TPU Nodes can have some quite unexpected behaviour for people who aren’t used to them! In particular, because the TPU is located on a physically different system to the machine you’re running your Python code on, your data cannot be local to your machine - any data pipeline that loads from your machine’s internal storage will totally fail! Instead, data must be stored in Google Cloud Storage where your data pipeline can still access it, even when the pipeline is running on the remote TPU node.

<Tip>

If you can fit all your data in memory as `np.ndarray` or `tf.Tensor`, then you can `fit()` on that data even when using Colab or a TPU Node, without needing to upload it to Google Cloud Storage.

</Tip>

<Tip>

**🤗Specific Hugging Face Tip🤗:** The methods `Dataset.to_tf_dataset()` and its higher-level wrapper `model.prepare_tf_dataset()` , which you will see throughout our TF code examples, will both fail on a TPU Node. The reason for this is that even though they create a `tf.data.Dataset` it is not a “pure” `tf.data` pipeline and uses `tf.numpy_function` or `Dataset.from_generator()` to stream data from the underlying HuggingFace `Dataset`. This HuggingFace `Dataset` is backed by data that is on a local disc and which the remote TPU Node will not be able to read.

</Tip>

The second way to access a TPU is via a **TPU VM.** When using a TPU VM, you connect directly to the machine that the TPU is attached to, much like training on a GPU VM. TPU VMs are generally easier to work with, particularly when it comes to your data pipeline. All of the above warnings do not apply to TPU VMs!

This is an opinionated document, so here’s our opinion: **Avoid using TPU Node if possible.


key: _node_content
string_value:
{"id_": "fde06c30-8b87-4a2e-a6d6-d1dbc82752c2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a137cf0-b5ba-4c86-a95b-56942746cfb7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4ca8acbf58680a9bfe46c25f65c3f91a76f25b2f43e27df163841119b4fad291", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "82599d5b-cfac-4a5e-ba7d-f16ddbe41f70", "node_type": "1", "metadata": {}, "hash": "e50e8902689a2231c6ea9c642a024754c791fa7623a555fa69bef0f87a71637b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1711, "end_char_idx": 4800, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In particular, because the TPU is located on a physically different system to the machine you’re running your Python code on, your data cannot be local to your machine - any data pipeline that loads from your machine’s internal storage will totally fail! Instead, data must be stored in Google Cloud Storage where your data pipeline can still access it, even when the pipeline is running on the remote TPU node.

<Tip>

If you can fit all your data in memory as `np.ndarray` or `tf.Tensor`, then you can `fit()` on that data even when using Colab or a TPU Node, without needing to upload it to Google Cloud Storage.

</Tip>

<Tip>

**🤗Specific Hugging Face Tip🤗:** The methods `Dataset.to_tf_dataset()` and its higher-level wrapper `model.prepare_tf_dataset()` , which you will see throughout our TF code examples, will both fail on a TPU Node. The reason for this is that even though they create a `tf.data.Dataset` it is not a “pure” `tf.data` pipeline and uses `tf.numpy_function` or `Dataset.from_generator()` to stream data from the underlying HuggingFace `Dataset`. This HuggingFace `Dataset` is backed by data that is on a local disc and which the remote TPU Node will not be able to read.

</Tip>

The second way to access a TPU is via a **TPU VM.** When using a TPU VM, you connect directly to the machine that the TPU is attached to, much like training on a GPU VM. TPU VMs are generally easier to work with, particularly when it comes to your data pipeline. All of the above warnings do not apply to TPU VMs!

This is an opinionated document, so here’s our opinion: **Avoid using TPU Node if possible.** It is more confusing and more difficult to debug than TPU VMs. It is also likely to be unsupported in future - Google’s latest TPU, TPUv4, can only be accessed as a TPU VM, which suggests that TPU Nodes are increasingly going to become a “legacy” access method. However, we understand that the only free TPU access is on Colab and Kaggle Kernels, which uses TPU Node - so we’ll try to explain how to handle it if you have to! Check the [TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) for code samples that explain this in more detail.

### What sizes of TPU are available?

A single TPU (a v2-8/v3-8/v4-8) runs 8 replicas. TPUs exist in **pods** that can run hundreds or thousands of replicas simultaneously. When you use more than a single TPU but less than a whole pod (for example, a v3-32), your TPU fleet is referred to as a **pod slice.**

When you access a free TPU via Colab, you generally get a single v2-8 TPU.

### I keep hearing about this XLA thing. What’s XLA, and how does it relate to TPUs?

XLA is an optimizing compiler, used by both TensorFlow and JAX. In JAX it is the only compiler, whereas in TensorFlow it is optional (but mandatory on TPU!). The easiest way to enable it when training a Keras model is to pass the argument `jit_compile=True` to `model.compile()`. If you don’t get any errors and performance is good, that’s a great sign that you’re ready to move to TPU!


key: _node_content
string_value:
{"id_": "82599d5b-cfac-4a5e-ba7d-f16ddbe41f70", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fde06c30-8b87-4a2e-a6d6-d1dbc82752c2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4911e6af3202b9165402b18b93c7998940e16e19c7a8ba47a11810b595b920df", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "48ae939f-8e40-4d00-a06e-b65a877efae5", "node_type": "1", "metadata": {}, "hash": "e0c77fa278d5c05fb22c5b5b9024535e7dbd512a5a7d187adf1c1ee610501ecc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3323, "end_char_idx": 6338, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
** It is more confusing and more difficult to debug than TPU VMs. It is also likely to be unsupported in future - Google’s latest TPU, TPUv4, can only be accessed as a TPU VM, which suggests that TPU Nodes are increasingly going to become a “legacy” access method. However, we understand that the only free TPU access is on Colab and Kaggle Kernels, which uses TPU Node - so we’ll try to explain how to handle it if you have to! Check the [TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) for code samples that explain this in more detail.

### What sizes of TPU are available?

A single TPU (a v2-8/v3-8/v4-8) runs 8 replicas. TPUs exist in **pods** that can run hundreds or thousands of replicas simultaneously. When you use more than a single TPU but less than a whole pod (for example, a v3-32), your TPU fleet is referred to as a **pod slice.**

When you access a free TPU via Colab, you generally get a single v2-8 TPU.

### I keep hearing about this XLA thing. What’s XLA, and how does it relate to TPUs?

XLA is an optimizing compiler, used by both TensorFlow and JAX. In JAX it is the only compiler, whereas in TensorFlow it is optional (but mandatory on TPU!). The easiest way to enable it when training a Keras model is to pass the argument `jit_compile=True` to `model.compile()`. If you don’t get any errors and performance is good, that’s a great sign that you’re ready to move to TPU!

Debugging on TPU is generally a bit harder than on CPU/GPU, so we recommend getting your code running on CPU/GPU with XLA first before trying it on TPU. You don’t have to train for long, of course - just for a few steps to make sure that your model and data pipeline are working like you expect them to.

<Tip>

XLA compiled code is usually faster - so even if you’re not planning to run on TPU, adding `jit_compile=True` can improve your performance. Be sure to note the caveats below about XLA compatibility, though!

</Tip>

<Tip warning={true}>

**Tip born of painful experience:** Although using `jit_compile=True` is a good way to get a speed boost and test if your CPU/GPU code is XLA-compatible, it can actually cause a lot of problems if you leave it in when actually training on TPU. XLA compilation will happen implicitly on TPU, so remember to remove that line before actually running your code on a TPU!

</Tip>

### How do I make my model XLA compatible?

In many cases, your code is probably XLA-compatible already! However, there are a few things that work in normal TensorFlow that don’t work in XLA. We’ve distilled them into three core rules below:

<Tip>

**🤗Specific HuggingFace Tip🤗:** We’ve put a lot of effort into rewriting our TensorFlow models and loss functions to be XLA-compatible. Our models and loss functions generally obey rule #1 and #2 by default, so you can skip over them if you’re using `transformers` models. Don’t forget about these rules when writing your own models and loss functions, though!


key: _node_content
string_value:
{"id_": "48ae939f-8e40-4d00-a06e-b65a877efae5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82599d5b-cfac-4a5e-ba7d-f16ddbe41f70", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "af83313da1d1862991138655f1d564470c24f58310f4916315502c054c01c675", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ea3724e-8404-4cd5-bc0e-604ebbb53fa6", "node_type": "1", "metadata": {}, "hash": "8b013ddea85fd54f6b6dfa37639c70ee7a4ae87425db59ff8eb0620cd7d8fc65", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4694, "end_char_idx": 7840, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you don’t get any errors and performance is good, that’s a great sign that you’re ready to move to TPU!

Debugging on TPU is generally a bit harder than on CPU/GPU, so we recommend getting your code running on CPU/GPU with XLA first before trying it on TPU. You don’t have to train for long, of course - just for a few steps to make sure that your model and data pipeline are working like you expect them to.

<Tip>

XLA compiled code is usually faster - so even if you’re not planning to run on TPU, adding `jit_compile=True` can improve your performance. Be sure to note the caveats below about XLA compatibility, though!

</Tip>

<Tip warning={true}>

**Tip born of painful experience:** Although using `jit_compile=True` is a good way to get a speed boost and test if your CPU/GPU code is XLA-compatible, it can actually cause a lot of problems if you leave it in when actually training on TPU. XLA compilation will happen implicitly on TPU, so remember to remove that line before actually running your code on a TPU!

</Tip>

### How do I make my model XLA compatible?

In many cases, your code is probably XLA-compatible already! However, there are a few things that work in normal TensorFlow that don’t work in XLA. We’ve distilled them into three core rules below:

<Tip>

**🤗Specific HuggingFace Tip🤗:** We’ve put a lot of effort into rewriting our TensorFlow models and loss functions to be XLA-compatible. Our models and loss functions generally obey rule #1 and #2 by default, so you can skip over them if you’re using `transformers` models. Don’t forget about these rules when writing your own models and loss functions, though!

</Tip>

#### XLA Rule #1: Your code cannot have “data-dependent conditionals”

What that means is that any `if` statement cannot depend on values inside a `tf.Tensor`. For example, this code block cannot be compiled with XLA!

```python
if tf.reduce_sum(tensor) > 10:
    tensor = tensor / 2.0
```

This might seem very restrictive at first, but most neural net code doesn’t need to do this. You can often get around this restriction by using `tf.cond` (see the documentation [here](https://www.tensorflow.org/api_docs/python/tf/cond)) or by removing the conditional and finding a clever math trick with indicator variables instead, like so:

```python
sum_over_10 = tf.cast(tf.reduce_sum(tensor) > 10, tf.float32)
tensor = tensor / (1.0 + sum_over_10)
```

This code has exactly the same effect as the code above, but by avoiding a conditional, we ensure it will compile with XLA without problems!

#### XLA Rule #2: Your code cannot have “data-dependent shapes”

What this means is that the shape of all of the `tf.Tensor` objects in your code cannot depend on their values. For example, the function `tf.unique` cannot be compiled with XLA, because it returns a `tensor` containing one instance of each unique value in the input. The shape of this output will obviously be different depending on how repetitive the input `Tensor` was, and so XLA refuses to handle it!

In general, most neural network code obeys rule #2 by default. However, there are a few common cases where it becomes a problem.


key: _node_content
string_value:
{"id_": "9ea3724e-8404-4cd5-bc0e-604ebbb53fa6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "48ae939f-8e40-4d00-a06e-b65a877efae5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b56620b114a803aa406473db411fceb4e291558ba6615c550ae4ddeb5aec197f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "38f6cb4d-f4d6-42b2-97d1-67e4ab57c7a5", "node_type": "1", "metadata": {}, "hash": "58230d26aa0812f7a2d4e8e8ad5753b88a7bca534ec119532e0f8f34be89ea9e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6251, "end_char_idx": 9602, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Don’t forget about these rules when writing your own models and loss functions, though!

</Tip>

#### XLA Rule #1: Your code cannot have “data-dependent conditionals”

What that means is that any `if` statement cannot depend on values inside a `tf.Tensor`. For example, this code block cannot be compiled with XLA!

```python
if tf.reduce_sum(tensor) > 10:
    tensor = tensor / 2.0
```

This might seem very restrictive at first, but most neural net code doesn’t need to do this. You can often get around this restriction by using `tf.cond` (see the documentation [here](https://www.tensorflow.org/api_docs/python/tf/cond)) or by removing the conditional and finding a clever math trick with indicator variables instead, like so:

```python
sum_over_10 = tf.cast(tf.reduce_sum(tensor) > 10, tf.float32)
tensor = tensor / (1.0 + sum_over_10)
```

This code has exactly the same effect as the code above, but by avoiding a conditional, we ensure it will compile with XLA without problems!

#### XLA Rule #2: Your code cannot have “data-dependent shapes”

What this means is that the shape of all of the `tf.Tensor` objects in your code cannot depend on their values. For example, the function `tf.unique` cannot be compiled with XLA, because it returns a `tensor` containing one instance of each unique value in the input. The shape of this output will obviously be different depending on how repetitive the input `Tensor` was, and so XLA refuses to handle it!

In general, most neural network code obeys rule #2 by default. However, there are a few common cases where it becomes a problem. One very common one is when you use **label masking**, setting your labels to a negative value to indicate that those positions should be ignored when computing the loss. If you look at NumPy or PyTorch loss functions that support label masking, you will often see code like this that uses [boolean indexing](https://numpy.org/doc/stable/user/basics.indexing.html#boolean-array-indexing):

```python
label_mask = labels >= 0
masked_outputs = outputs[label_mask]
masked_labels = labels[label_mask]
loss = compute_loss(masked_outputs, masked_labels)
mean_loss = torch.mean(loss)
```

This code is totally fine in NumPy or PyTorch, but it breaks in XLA! Why? Because the shape of `masked_outputs` and `masked_labels` depends on how many positions are masked - that makes it a **data-dependent shape.** However, just like for rule #1, we can often rewrite this code to yield exactly the same output without any data-dependent shapes.

```python
label_mask = tf.cast(labels >= 0, tf.float32)
loss = compute_loss(outputs, labels)
loss = loss * label_mask  # Set negative label positions to 0
mean_loss = tf.reduce_sum(loss) / tf.reduce_sum(label_mask)
```

Here, we avoid data-dependent shapes by computing the loss for every position, but zeroing out the masked positions in both the numerator and denominator when we calculate the mean, which yields exactly the same result as the first block while maintaining XLA compatibility. Note that we use the same trick as in rule #1 - converting a `tf.bool` to `tf.float32` and using it as an indicator variable. This is a really useful trick, so remember it if you need to convert your own code to XLA!

#### XLA Rule #3: XLA will need to recompile your model for every different input shape it sees

This is the big one.


key: _node_content
string_value:
{"id_": "38f6cb4d-f4d6-42b2-97d1-67e4ab57c7a5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ea3724e-8404-4cd5-bc0e-604ebbb53fa6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "90df39d07433e274e49a5599e0b2023fca069a714b22960626a3368835a3bb39", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed9cb157-3176-47f0-98af-115e7235791c", "node_type": "1", "metadata": {}, "hash": "bceda79f31cca7fc833bfbee6c3cd792c41f7c236242780e746b2fac36212354", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8012, "end_char_idx": 11285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you look at NumPy or PyTorch loss functions that support label masking, you will often see code like this that uses [boolean indexing](https://numpy.org/doc/stable/user/basics.indexing.html#boolean-array-indexing):

```python
label_mask = labels >= 0
masked_outputs = outputs[label_mask]
masked_labels = labels[label_mask]
loss = compute_loss(masked_outputs, masked_labels)
mean_loss = torch.mean(loss)
```

This code is totally fine in NumPy or PyTorch, but it breaks in XLA! Why? Because the shape of `masked_outputs` and `masked_labels` depends on how many positions are masked - that makes it a **data-dependent shape.** However, just like for rule #1, we can often rewrite this code to yield exactly the same output without any data-dependent shapes.

```python
label_mask = tf.cast(labels >= 0, tf.float32)
loss = compute_loss(outputs, labels)
loss = loss * label_mask  # Set negative label positions to 0
mean_loss = tf.reduce_sum(loss) / tf.reduce_sum(label_mask)
```

Here, we avoid data-dependent shapes by computing the loss for every position, but zeroing out the masked positions in both the numerator and denominator when we calculate the mean, which yields exactly the same result as the first block while maintaining XLA compatibility. Note that we use the same trick as in rule #1 - converting a `tf.bool` to `tf.float32` and using it as an indicator variable. This is a really useful trick, so remember it if you need to convert your own code to XLA!

#### XLA Rule #3: XLA will need to recompile your model for every different input shape it sees

This is the big one. What this means is that if your input shapes are very variable, XLA will have to recompile your model over and over, which will create huge performance problems. This commonly arises in NLP models, where input texts have variable lengths after tokenization. In other modalities, static shapes are more common and this rule is much less of a problem.

How can you get around rule #3? The key is **padding** - if you pad all your inputs to the same length, and then use an `attention_mask`, you can get the same results as you’d get from variable shapes, but without any XLA issues. However, excessive padding can cause severe slowdown too - if you pad all your samples to the maximum length in the whole dataset, you might end up with batches consisting endless padding tokens, which will waste a lot of compute and memory!

There isn’t a perfect solution to this problem. However, you can try some tricks. One very useful trick is to **pad batches of samples up to a multiple of a number like 32 or 64 tokens.** This often only increases the number of tokens by a small amount, but it hugely reduces the number of unique input shapes, because every input shape now has to be a multiple of 32 or 64. Fewer unique input shapes means fewer XLA compilations!

<Tip>

**🤗Specific HuggingFace Tip🤗:** Our tokenizers and data collators have methods that can help you here. You can use `padding="max_length"` or `padding="longest"` when calling tokenizers to get them to output padded data. Our tokenizers and data collators also have a `pad_to_multiple_of` argument that you can use to reduce the number of unique input shapes you see!

</Tip>

### How do I actually train my model on TPU?


key: _node_content
string_value:
{"id_": "ed9cb157-3176-47f0-98af-115e7235791c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "38f6cb4d-f4d6-42b2-97d1-67e4ab57c7a5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cea4f346c438ead48ef6b705ce5de2de3fed5877667ed2b32b01af457765c2b5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b6179de9-4d54-4970-973c-405a6ea83f05", "node_type": "1", "metadata": {}, "hash": "f8eba4a6ba0f7f20c573ecc5e00d4e1fc61d30b1e627ac6a81006893a89e2db5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9603, "end_char_idx": 12855, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
What this means is that if your input shapes are very variable, XLA will have to recompile your model over and over, which will create huge performance problems. This commonly arises in NLP models, where input texts have variable lengths after tokenization. In other modalities, static shapes are more common and this rule is much less of a problem.

How can you get around rule #3? The key is **padding** - if you pad all your inputs to the same length, and then use an `attention_mask`, you can get the same results as you’d get from variable shapes, but without any XLA issues. However, excessive padding can cause severe slowdown too - if you pad all your samples to the maximum length in the whole dataset, you might end up with batches consisting endless padding tokens, which will waste a lot of compute and memory!

There isn’t a perfect solution to this problem. However, you can try some tricks. One very useful trick is to **pad batches of samples up to a multiple of a number like 32 or 64 tokens.** This often only increases the number of tokens by a small amount, but it hugely reduces the number of unique input shapes, because every input shape now has to be a multiple of 32 or 64. Fewer unique input shapes means fewer XLA compilations!

<Tip>

**🤗Specific HuggingFace Tip🤗:** Our tokenizers and data collators have methods that can help you here. You can use `padding="max_length"` or `padding="longest"` when calling tokenizers to get them to output padded data. Our tokenizers and data collators also have a `pad_to_multiple_of` argument that you can use to reduce the number of unique input shapes you see!

</Tip>

### How do I actually train my model on TPU?

Once your training is XLA-compatible and (if you’re using TPU Node / Colab) your dataset has been prepared appropriately, running on TPU is surprisingly easy! All you really need to change in your code is to add a few lines to initialize your TPU, and to ensure that your model and dataset are created inside a `TPUStrategy` scope. Take a look at [our TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) to see this in action!

### Summary

There was a lot in here, so let’s summarize with a quick checklist you can follow when you want to get your model ready for TPU training:

- Make sure your code follows the three rules of XLA
- Compile your model with `jit_compile=True` on CPU/GPU and confirm that you can train it with XLA
- Either load your dataset into memory or use a TPU-compatible dataset loading approach (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Migrate your code either to Colab (with accelerator set to “TPU”) or a TPU VM on Google Cloud
- Add TPU initializer code (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Create your `TPUStrategy` and make sure dataset loading and model creation are inside the `strategy.scope()` (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Don’t forget to take `jit_compile=True` out again when you move to TPU!


key: _node_content
string_value:
{"id_": "b6179de9-4d54-4970-973c-405a6ea83f05", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ed9cb157-3176-47f0-98af-115e7235791c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "25cd77b7f8ff545b14802fa6f7ef702821338744c78a817cd2794eceaeef6473", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11287, "end_char_idx": 12897, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Once your training is XLA-compatible and (if you’re using TPU Node / Colab) your dataset has been prepared appropriately, running on TPU is surprisingly easy! All you really need to change in your code is to add a few lines to initialize your TPU, and to ensure that your model and dataset are created inside a `TPUStrategy` scope. Take a look at [our TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) to see this in action!

### Summary

There was a lot in here, so let’s summarize with a quick checklist you can follow when you want to get your model ready for TPU training:

- Make sure your code follows the three rules of XLA
- Compile your model with `jit_compile=True` on CPU/GPU and confirm that you can train it with XLA
- Either load your dataset into memory or use a TPU-compatible dataset loading approach (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Migrate your code either to Colab (with accelerator set to “TPU”) or a TPU VM on Google Cloud
- Add TPU initializer code (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Create your `TPUStrategy` and make sure dataset loading and model creation are inside the `strategy.scope()` (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Don’t forget to take `jit_compile=True` out again when you move to TPU!
- 🙏🙏🙏🥺🥺🥺
- Call model.fit()
- You did it!


key: _node_content
string_value:
{"id_": "c0a798ad-d97d-4727-8bee-64cf906488f5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "19cacafc-b25c-4d5b-9268-b920de77d67d", "node_type": "1", "metadata": {}, "hash": "2f4bcf3a09487322d635c84899dffdb6348017b7807ed1898cc5f0d8429de56f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4461, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Quick tour

[[open-in-colab]]

Get up and running with 🤗 Transformers! Whether you're a developer or an everyday user, this quick tour will help you get started and show you how to use the [`pipeline`] for inference, load a pretrained model and preprocessor with an [AutoClass](./model_doc/auto), and quickly train a model with PyTorch or TensorFlow. If you're a beginner, we recommend checking out our tutorials or [course](https://huggingface.co/course/chapter1/1) next for more in-depth explanations of the concepts introduced here.

Before you begin, make sure you have all the necessary libraries installed:

```bash
!pip install transformers datasets evaluate accelerate
```

You'll also need to install your preferred machine learning framework:

<frameworkcontent>
<pt>

```bash
pip install torch
```
</pt>
<tf>

```bash
pip install tensorflow
```
</tf>
</frameworkcontent>

## Pipeline

<Youtube id="tiZFewofSLM"/>

The [`pipeline`] is the easiest and fastest way to use a pretrained model for inference. You can use the [`pipeline`] out-of-the-box for many tasks across different modalities, some of which are shown in the table below:

<Tip>

For a complete list of available tasks, check out the [pipeline API reference](./main_classes/pipelines).

</Tip>

| **Task**                     | **Description**                                                                                              | **Modality**    | **Pipeline identifier**                       |
|------------------------------|--------------------------------------------------------------------------------------------------------------|-----------------|-----------------------------------------------|
| Text classification          | assign a label to a given sequence of text                                                                   | NLP             | pipeline(task=“sentiment-analysis”)           |
| Text generation              | generate text given a prompt                                                                                 | NLP             | pipeline(task=“text-generation”)              |
| Summarization                | generate a summary of a sequence of text or document                                                         | NLP             | pipeline(task=“summarization”)                |
| Image classification         | assign a label to an image                                                                                   | Computer vision | pipeline(task=“image-classification”)         |
| Image segmentation           | assign a label to each individual pixel of an image (supports semantic, panoptic, and instance segmentation) | Computer vision | pipeline(task=“image-segmentation”)           |
| Object detection             | predict the bounding boxes and classes of objects in an image                                                | Computer vision | pipeline(task=“object-detection”)             |
| Audio classification         | assign a label to some audio data                                                                            | Audio           | pipeline(task=“audio-classification”)         |
| Automatic speech recognition | transcribe speech into text                                                                                  | Audio           | pipeline(task=“automatic-speech-recognition”) |
| Visual question answering    | answer a question about the image, given an image and a question                                             | Multimodal      | pipeline(task=“vqa”)                          |
| Document question answering  | answer a question about the document, given a document and a question                                        | Multimodal      | pipeline(task="document-question-answering")  |
| Image captioning             | generate a caption for a given image                                                                         | Multimodal      | pipeline(task="image-to-text")                |

Start by creating an instance of [`pipeline`] and specifying a task you want to use it for. In this guide, you'll use the [`pipeline`] for sentiment analysis as an example:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("sentiment-analysis")
```

The [`pipeline`] downloads and caches a default [pretrained model](https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english) and tokenizer for sentiment analysis.


key: _node_content
string_value:
{"id_": "19cacafc-b25c-4d5b-9268-b920de77d67d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c0a798ad-d97d-4727-8bee-64cf906488f5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cd8be0b66c53cc8930b55d924306482f9592a86448d971cde3a7f3e9151a3cbb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a3ba50bf-f05f-4156-964a-8e133d5af31a", "node_type": "1", "metadata": {}, "hash": "32067a2c374b2adeb814b4ffbebeffbf7c02b7ab29a8600dafdd1ceee5d13045", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4094, "end_char_idx": 6199, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this guide, you'll use the [`pipeline`] for sentiment analysis as an example:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("sentiment-analysis")
```

The [`pipeline`] downloads and caches a default [pretrained model](https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english) and tokenizer for sentiment analysis. Now you can use the `classifier` on your target text:

```py
>>> classifier("We are very happy to show you the 🤗 Transformers library.")
[{'label': 'POSITIVE', 'score': 0.9998}]
```

If you have more than one input, pass your inputs as a list to the [`pipeline`] to return a list of dictionaries:

```py
>>> results = classifier(["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."])
>>> for result in results:
...     print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
label: POSITIVE, with score: 0.9998
label: NEGATIVE, with score: 0.5309
```

The [`pipeline`] can also iterate over an entire dataset for any task you like. For this example, let's choose automatic speech recognition as our task:

```py
>>> import torch
>>> from transformers import pipeline

>>> speech_recognizer = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-base-960h")
```

Load an audio dataset (see the 🤗 Datasets [Quick Start](https://huggingface.co/docs/datasets/quickstart#audio) for more details) you'd like to iterate over. For example, load the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")  # doctest: +IGNORE_RESULT
```

You need to make sure the sampling rate of the dataset matches the sampling 
rate [`facebook/wav2vec2-base-960h`](https://huggingface.co/facebook/wav2vec2-base-960h) was trained on:

```py
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=speech_recognizer.feature_extractor.sampling_rate))
```

The audio files are automatically loaded and resampled when calling the `"audio"` column.


key: _node_content
string_value:
{"id_": "a3ba50bf-f05f-4156-964a-8e133d5af31a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "19cacafc-b25c-4d5b-9268-b920de77d67d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bd177c37a7dd2f1057a114729cda3c76012e491bbc46ceb6053ed0e4bc4248bc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25ba36cf-99b8-41c1-9ef5-348a4fe9cffd", "node_type": "1", "metadata": {}, "hash": "edf7908e6722d5c497727af26f915dbb5a8f1176cf8766e11f4db6119f4a7f50", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4850, "end_char_idx": 7514, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
", "We hope you don't hate it."])
>>> for result in results:
...     print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
label: POSITIVE, with score: 0.9998
label: NEGATIVE, with score: 0.5309
```

The [`pipeline`] can also iterate over an entire dataset for any task you like. For this example, let's choose automatic speech recognition as our task:

```py
>>> import torch
>>> from transformers import pipeline

>>> speech_recognizer = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-base-960h")
```

Load an audio dataset (see the 🤗 Datasets [Quick Start](https://huggingface.co/docs/datasets/quickstart#audio) for more details) you'd like to iterate over. For example, load the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")  # doctest: +IGNORE_RESULT
```

You need to make sure the sampling rate of the dataset matches the sampling 
rate [`facebook/wav2vec2-base-960h`](https://huggingface.co/facebook/wav2vec2-base-960h) was trained on:

```py
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=speech_recognizer.feature_extractor.sampling_rate))
```

The audio files are automatically loaded and resampled when calling the `"audio"` column.
Extract the raw waveform arrays from the first 4 samples and pass it as a list to the pipeline:

```py
>>> result = speech_recognizer(dataset[:4]["audio"])
>>> print([d["text"] for d in result])
['I WOULD LIKE TO SET UP A JOINT ACCOUNT WITH MY PARTNER HOW DO I PROCEED WITH DOING THAT', "FONDERING HOW I'D SET UP A JOIN TO HELL T WITH MY WIFE AND WHERE THE AP MIGHT BE", "I I'D LIKE TOY SET UP A JOINT ACCOUNT WITH MY PARTNER I'M NOT SEEING THE OPTION TO DO IT ON THE APSO I CALLED IN TO GET SOME HELP CAN I JUST DO IT OVER THE PHONE WITH YOU AND GIVE YOU THE INFORMATION OR SHOULD I DO IT IN THE AP AN I'M MISSING SOMETHING UQUETTE HAD PREFERRED TO JUST DO IT OVER THE PHONE OF POSSIBLE THINGS", 'HOW DO I FURN A JOINA COUT']
```

For larger datasets where the inputs are big (like in speech or vision), you'll want to pass a generator instead of a list to load all the inputs in memory. Take a look at the [pipeline API reference](./main_classes/pipelines) for more information.

### Use another model and tokenizer in the pipeline

The [`pipeline`] can accommodate any model from the [Hub](https://huggingface.co/models), making it easy to adapt the [`pipeline`] for other use-cases. For example, if you'd like a model capable of handling French text, use the tags on the Hub to filter for an appropriate model.


key: _node_content
string_value:
{"id_": "25ba36cf-99b8-41c1-9ef5-348a4fe9cffd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a3ba50bf-f05f-4156-964a-8e133d5af31a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7e645fd1a6c69de3e1ca9e1169b918abb9ca8a6411728282534304b12d9cd207", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c3de771-63e3-40a0-ab42-d4cfc3ad25de", "node_type": "1", "metadata": {}, "hash": "6d2f53bb2cf2f564603ea8e51ba990c158529239531bd87c96707ac5af69e07c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6200, "end_char_idx": 9335, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Extract the raw waveform arrays from the first 4 samples and pass it as a list to the pipeline:

```py
>>> result = speech_recognizer(dataset[:4]["audio"])
>>> print([d["text"] for d in result])
['I WOULD LIKE TO SET UP A JOINT ACCOUNT WITH MY PARTNER HOW DO I PROCEED WITH DOING THAT', "FONDERING HOW I'D SET UP A JOIN TO HELL T WITH MY WIFE AND WHERE THE AP MIGHT BE", "I I'D LIKE TOY SET UP A JOINT ACCOUNT WITH MY PARTNER I'M NOT SEEING THE OPTION TO DO IT ON THE APSO I CALLED IN TO GET SOME HELP CAN I JUST DO IT OVER THE PHONE WITH YOU AND GIVE YOU THE INFORMATION OR SHOULD I DO IT IN THE AP AN I'M MISSING SOMETHING UQUETTE HAD PREFERRED TO JUST DO IT OVER THE PHONE OF POSSIBLE THINGS", 'HOW DO I FURN A JOINA COUT']
```

For larger datasets where the inputs are big (like in speech or vision), you'll want to pass a generator instead of a list to load all the inputs in memory. Take a look at the [pipeline API reference](./main_classes/pipelines) for more information.

### Use another model and tokenizer in the pipeline

The [`pipeline`] can accommodate any model from the [Hub](https://huggingface.co/models), making it easy to adapt the [`pipeline`] for other use-cases. For example, if you'd like a model capable of handling French text, use the tags on the Hub to filter for an appropriate model. The top filtered result returns a multilingual [BERT model](https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment) finetuned for sentiment analysis you can use for French text:

```py
>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
```

<frameworkcontent>
<pt>
Use [`AutoModelForSequenceClassification`] and [`AutoTokenizer`] to load the pretrained model and it's associated tokenizer (more on an `AutoClass` in the next section):

```py
>>> from transformers import AutoTokenizer, AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained(model_name)
>>> tokenizer = AutoTokenizer.from_pretrained(model_name)
```
</pt>
<tf>
Use [`TFAutoModelForSequenceClassification`] and [`AutoTokenizer`] to load the pretrained model and it's associated tokenizer (more on an `TFAutoClass` in the next section):

```py
>>> from transformers import AutoTokenizer, TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained(model_name)
>>> tokenizer = AutoTokenizer.from_pretrained(model_name)
```
</tf>
</frameworkcontent>

Specify the model and tokenizer in the [`pipeline`], and now you can apply the `classifier` on French text:

```py
>>> classifier = pipeline("sentiment-analysis", model=model, tokenizer=tokenizer)
>>> classifier("Nous sommes très heureux de vous présenter la bibliothèque 🤗 Transformers.")
[{'label': '5 stars', 'score': 0.7273}]
```

If you can't find a model for your use-case, you'll need to finetune a pretrained model on your data. Take a look at our [finetuning tutorial](./training) to learn how. Finally, after you've finetuned your pretrained model, please consider [sharing](./model_sharing) the model with the community on the Hub to democratize machine learning for everyone!


key: _node_content
string_value:
{"id_": "3c3de771-63e3-40a0-ab42-d4cfc3ad25de", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "25ba36cf-99b8-41c1-9ef5-348a4fe9cffd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "336b20c14803a0331e5df2436f4a51455cda01f666fa1b4a6fb0081508af5b10", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03fe8ad3-05f4-4206-a2c0-8f526e808a74", "node_type": "1", "metadata": {}, "hash": "72d817d147d8d680bdfe0ddc949cc311d091d9696f1860abdac2c976400ec12a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8938, "end_char_idx": 11779, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[{'label': '5 stars', 'score': 0.7273}]
```

If you can't find a model for your use-case, you'll need to finetune a pretrained model on your data. Take a look at our [finetuning tutorial](./training) to learn how. Finally, after you've finetuned your pretrained model, please consider [sharing](./model_sharing) the model with the community on the Hub to democratize machine learning for everyone! 🤗

## AutoClass

<Youtube id="AhChOFRegn4"/>

Under the hood, the [`AutoModelForSequenceClassification`] and [`AutoTokenizer`] classes work together to power the [`pipeline`] you used above. An [AutoClass](./model_doc/auto) is a shortcut that automatically retrieves the architecture of a pretrained model from its name or path. You only need to select the appropriate `AutoClass` for your task and it's associated preprocessing class. 

Let's return to the example from the previous section and see how you can use the `AutoClass` to replicate the results of the [`pipeline`].

### AutoTokenizer

A tokenizer is responsible for preprocessing text into an array of numbers as inputs to a model. There are multiple rules that govern the tokenization process, including how to split a word and at what level words should be split (learn more about tokenization in the [tokenizer summary](./tokenizer_summary)). The most important thing to remember is you need to instantiate a tokenizer with the same model name to ensure you're using the same tokenization rules a model was pretrained with.

Load a tokenizer with [`AutoTokenizer`]:

```py
>>> from transformers import AutoTokenizer

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> tokenizer = AutoTokenizer.from_pretrained(model_name)
```

Pass your text to the tokenizer:

```py
>>> encoding = tokenizer("We are very happy to show you the 🤗 Transformers library.")
>>> print(encoding)
{'input_ids': [101, 11312, 10320, 12495, 19308, 10114, 11391, 10855, 10103, 100, 58263, 13299, 119, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

The tokenizer returns a dictionary containing:

* [input_ids](./glossary#input-ids): numerical representations of your tokens.
* [attention_mask](./glossary#attention-mask): indicates which tokens should be attended to.

A tokenizer can also accept a list of inputs, and pad and truncate the text to return a batch with uniform length:

<frameworkcontent>
<pt>

```py
>>> pt_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="pt",
... )
```
</pt>
<tf>

```py
>>> tf_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it.


key: _node_content
string_value:
{"id_": "03fe8ad3-05f4-4206-a2c0-8f526e808a74", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c3de771-63e3-40a0-ab42-d4cfc3ad25de", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e9b0118e2e0adcb3df1d073ab704429b508c0b9e274ee9bbde921612ac3818bb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2f0cd4e4-c3bc-465c-86c7-f6c2b6263ada", "node_type": "1", "metadata": {}, "hash": "4bda205eb754f0a8adfb712a013aba8e32a115c1437d23bfd32022cf56b9aed6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10776, "end_char_idx": 13097, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> print(encoding)
{'input_ids': [101, 11312, 10320, 12495, 19308, 10114, 11391, 10855, 10103, 100, 58263, 13299, 119, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

The tokenizer returns a dictionary containing:

* [input_ids](./glossary#input-ids): numerical representations of your tokens.
* [attention_mask](./glossary#attention-mask): indicates which tokens should be attended to.

A tokenizer can also accept a list of inputs, and pad and truncate the text to return a batch with uniform length:

<frameworkcontent>
<pt>

```py
>>> pt_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="pt",
... )
```
</pt>
<tf>

```py
>>> tf_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="tf",
... )
```
</tf>
</frameworkcontent>

<Tip>

Check out the [preprocess](./preprocessing) tutorial for more details about tokenization, and how to use an [`AutoImageProcessor`], [`AutoFeatureExtractor`] and [`AutoProcessor`] to preprocess image, audio, and multimodal inputs.

</Tip>

### AutoModel

<frameworkcontent>
<pt>
🤗 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`AutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`AutoModel`] for the task. For text (or sequence) classification, you should load [`AutoModelForSequenceClassification`]:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> pt_model = AutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You just have to unpack the dictionary by adding `**`:

```py
>>> pt_outputs = pt_model(**pt_batch)
```

The model outputs the final activations in the `logits` attribute.


key: _node_content
string_value:
{"id_": "2f0cd4e4-c3bc-465c-86c7-f6c2b6263ada", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "03fe8ad3-05f4-4206-a2c0-8f526e808a74", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3513a1ce50218dd02356ea83a236c6b0fb65a1230620db3237c1c66f73d64184", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "22e1d760-de46-41cb-8059-7aea43ba9829", "node_type": "1", "metadata": {}, "hash": "fd600be74cf70414fd62fd10e349dd281905975cbed45424c415d31b56104e78", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11494, "end_char_idx": 14322, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="pt",
... )
```
</pt>
<tf>

```py
>>> tf_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="tf",
... )
```
</tf>
</frameworkcontent>

<Tip>

Check out the [preprocess](./preprocessing) tutorial for more details about tokenization, and how to use an [`AutoImageProcessor`], [`AutoFeatureExtractor`] and [`AutoProcessor`] to preprocess image, audio, and multimodal inputs.

</Tip>

### AutoModel

<frameworkcontent>
<pt>
🤗 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`AutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`AutoModel`] for the task. For text (or sequence) classification, you should load [`AutoModelForSequenceClassification`]:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> pt_model = AutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You just have to unpack the dictionary by adding `**`:

```py
>>> pt_outputs = pt_model(**pt_batch)
```

The model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:

```py
>>> from torch import nn

>>> pt_predictions = nn.functional.softmax(pt_outputs.logits, dim=-1)
>>> print(pt_predictions)
tensor([[0.0021, 0.0018, 0.0115, 0.2121, 0.7725],
        [0.2084, 0.1826, 0.1969, 0.1755, 0.2365]], grad_fn=<SoftmaxBackward0>)
```
</pt>
<tf>
🤗 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`TFAutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`TFAutoModel`] for the task. For text (or sequence) classification, you should load [`TFAutoModelForSequenceClassification`]:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You can pass the tensors as-is:

```py
>>> tf_outputs = tf_model(tf_batch)
```

The model outputs the final activations in the `logits` attribute.


key: _node_content
string_value:
{"id_": "22e1d760-de46-41cb-8059-7aea43ba9829", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2f0cd4e4-c3bc-465c-86c7-f6c2b6263ada", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a91be36d14f078ccb78b9495b2704a80321535abc03244a5f2dc4db906a19a16", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "45d23c0d-c060-4633-b0d4-9b63837d6165", "node_type": "1", "metadata": {}, "hash": "994a8fa4f2c43b35b4b170b5cfd214a12b0b64a18abe435becdaea554f2c7cc5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12852, "end_char_idx": 15017, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

Now pass your preprocessed batch of inputs directly to the model. You just have to unpack the dictionary by adding `**`:

```py
>>> pt_outputs = pt_model(**pt_batch)
```

The model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:

```py
>>> from torch import nn

>>> pt_predictions = nn.functional.softmax(pt_outputs.logits, dim=-1)
>>> print(pt_predictions)
tensor([[0.0021, 0.0018, 0.0115, 0.2121, 0.7725],
        [0.2084, 0.1826, 0.1969, 0.1755, 0.2365]], grad_fn=<SoftmaxBackward0>)
```
</pt>
<tf>
🤗 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`TFAutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`TFAutoModel`] for the task. For text (or sequence) classification, you should load [`TFAutoModelForSequenceClassification`]:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You can pass the tensors as-is:

```py
>>> tf_outputs = tf_model(tf_batch)
```

The model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:

```py
>>> import tensorflow as tf

>>> tf_predictions = tf.nn.softmax(tf_outputs.logits, axis=-1)
>>> tf_predictions  # doctest: +IGNORE_RESULT
```
</tf>
</frameworkcontent>

<Tip>

All 🤗 Transformers models (PyTorch or TensorFlow) output the tensors *before* the final activation
function (like softmax) because the final activation function is often fused with the loss. Model outputs are special dataclasses so their attributes are autocompleted in an IDE. The model outputs behave like a tuple or a dictionary (you can index with an integer, a slice or a string) in which case, attributes that are None are ignored.


key: _node_content
string_value:
{"id_": "45d23c0d-c060-4633-b0d4-9b63837d6165", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "22e1d760-de46-41cb-8059-7aea43ba9829", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "739c0888f3584dd8e5d8cc3fb3152ee96945dd133b2fa895647bfcfb51e84ab7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c09b860d-4b95-4f92-a8a9-b6a49c0d97f9", "node_type": "1", "metadata": {}, "hash": "2baf2fec40273bdf1b3868a72f659f29affd0f09d5161aaa179d466f2c429ccb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13524, "end_char_idx": 16219, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means you can load an [`TFAutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`TFAutoModel`] for the task. For text (or sequence) classification, you should load [`TFAutoModelForSequenceClassification`]:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You can pass the tensors as-is:

```py
>>> tf_outputs = tf_model(tf_batch)
```

The model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:

```py
>>> import tensorflow as tf

>>> tf_predictions = tf.nn.softmax(tf_outputs.logits, axis=-1)
>>> tf_predictions  # doctest: +IGNORE_RESULT
```
</tf>
</frameworkcontent>

<Tip>

All 🤗 Transformers models (PyTorch or TensorFlow) output the tensors *before* the final activation
function (like softmax) because the final activation function is often fused with the loss. Model outputs are special dataclasses so their attributes are autocompleted in an IDE. The model outputs behave like a tuple or a dictionary (you can index with an integer, a slice or a string) in which case, attributes that are None are ignored.

</Tip>

### Save a model

<frameworkcontent>
<pt>
Once your model is fine-tuned, you can save it with its tokenizer using [`PreTrainedModel.save_pretrained`]:

```py
>>> pt_save_directory = "./pt_save_pretrained"
>>> tokenizer.save_pretrained(pt_save_directory)  # doctest: +IGNORE_RESULT
>>> pt_model.save_pretrained(pt_save_directory)
```

When you are ready to use the model again, reload it with [`PreTrainedModel.from_pretrained`]:

```py
>>> pt_model = AutoModelForSequenceClassification.from_pretrained("./pt_save_pretrained")
```
</pt>
<tf>
Once your model is fine-tuned, you can save it with its tokenizer using [`TFPreTrainedModel.save_pretrained`]:

```py
>>> tf_save_directory = "./tf_save_pretrained"
>>> tokenizer.save_pretrained(tf_save_directory)  # doctest: +IGNORE_RESULT
>>> tf_model.save_pretrained(tf_save_directory)
```

When you are ready to use the model again, reload it with [`TFPreTrainedModel.from_pretrained`]:

```py
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained("./tf_save_pretrained")
```
</tf>
</frameworkcontent>

One particularly cool 🤗 Transformers feature is the ability to save a model and reload it as either a PyTorch or TensorFlow model.


key: _node_content
string_value:
{"id_": "c09b860d-4b95-4f92-a8a9-b6a49c0d97f9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45d23c0d-c060-4633-b0d4-9b63837d6165", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5c982ad8551c417b6b4cf331d16215cbc608cbb0faace1de4e5950fe8ae9fe68", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "18b1306c-48a5-4818-81d0-d66f73942fa4", "node_type": "1", "metadata": {}, "hash": "b428c2405a58a5cbe5d38fd98ca9ac2d25e96810c349ffee301f0b22621ed4c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14771, "end_char_idx": 18154, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Model outputs are special dataclasses so their attributes are autocompleted in an IDE. The model outputs behave like a tuple or a dictionary (you can index with an integer, a slice or a string) in which case, attributes that are None are ignored.

</Tip>

### Save a model

<frameworkcontent>
<pt>
Once your model is fine-tuned, you can save it with its tokenizer using [`PreTrainedModel.save_pretrained`]:

```py
>>> pt_save_directory = "./pt_save_pretrained"
>>> tokenizer.save_pretrained(pt_save_directory)  # doctest: +IGNORE_RESULT
>>> pt_model.save_pretrained(pt_save_directory)
```

When you are ready to use the model again, reload it with [`PreTrainedModel.from_pretrained`]:

```py
>>> pt_model = AutoModelForSequenceClassification.from_pretrained("./pt_save_pretrained")
```
</pt>
<tf>
Once your model is fine-tuned, you can save it with its tokenizer using [`TFPreTrainedModel.save_pretrained`]:

```py
>>> tf_save_directory = "./tf_save_pretrained"
>>> tokenizer.save_pretrained(tf_save_directory)  # doctest: +IGNORE_RESULT
>>> tf_model.save_pretrained(tf_save_directory)
```

When you are ready to use the model again, reload it with [`TFPreTrainedModel.from_pretrained`]:

```py
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained("./tf_save_pretrained")
```
</tf>
</frameworkcontent>

One particularly cool 🤗 Transformers feature is the ability to save a model and reload it as either a PyTorch or TensorFlow model. The `from_pt` or `from_tf` parameter can convert the model from one framework to the other:

<frameworkcontent>
<pt>

```py
>>> from transformers import AutoModel

>>> tokenizer = AutoTokenizer.from_pretrained(tf_save_directory)
>>> pt_model = AutoModelForSequenceClassification.from_pretrained(tf_save_directory, from_tf=True)
```
</pt>
<tf>

```py
>>> from transformers import TFAutoModel

>>> tokenizer = AutoTokenizer.from_pretrained(pt_save_directory)
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(pt_save_directory, from_pt=True)
```
</tf>
</frameworkcontent>

## Custom model builds

You can modify the model's configuration class to change how a model is built. The configuration specifies a model's attributes, such as the number of hidden layers or attention heads. You start from scratch when you initialize a model from a custom configuration class. The model attributes are randomly initialized, and you'll need to train the model before you can use it to get meaningful results.

Start by importing [`AutoConfig`], and then load the pretrained model you want to modify. Within [`AutoConfig.from_pretrained`], you can specify the attribute you want to change, such as the number of attention heads:

```py
>>> from transformers import AutoConfig

>>> my_config = AutoConfig.from_pretrained("distilbert/distilbert-base-uncased", n_heads=12)
```

<frameworkcontent>
<pt>
Create a model from your custom configuration with [`AutoModel.from_config`]:

```py
>>> from transformers import AutoModel

>>> my_model = AutoModel.from_config(my_config)
```
</pt>
<tf>
Create a model from your custom configuration with [`TFAutoModel.from_config`]:

```py
>>> from transformers import TFAutoModel

>>> my_model = TFAutoModel.from_config(my_config)
```
</tf>
</frameworkcontent>

Take a look at the [Create a custom architecture](./create_a_model) guide for more information about building custom configurations.


key: _node_content
string_value:
{"id_": "18b1306c-48a5-4818-81d0-d66f73942fa4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c09b860d-4b95-4f92-a8a9-b6a49c0d97f9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d5aa84a94af2ee233166e26e748d94ca547122d7f56b02b638dbd1607e38a086", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d82282e1-6b50-4a00-b469-f031e04d7f50", "node_type": "1", "metadata": {}, "hash": "75e61de65b9b3126c2f3b0f785aeae98034f23cbc1c8198e6faa886ec8b6c9a3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16912, "end_char_idx": 20239, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The configuration specifies a model's attributes, such as the number of hidden layers or attention heads. You start from scratch when you initialize a model from a custom configuration class. The model attributes are randomly initialized, and you'll need to train the model before you can use it to get meaningful results.

Start by importing [`AutoConfig`], and then load the pretrained model you want to modify. Within [`AutoConfig.from_pretrained`], you can specify the attribute you want to change, such as the number of attention heads:

```py
>>> from transformers import AutoConfig

>>> my_config = AutoConfig.from_pretrained("distilbert/distilbert-base-uncased", n_heads=12)
```

<frameworkcontent>
<pt>
Create a model from your custom configuration with [`AutoModel.from_config`]:

```py
>>> from transformers import AutoModel

>>> my_model = AutoModel.from_config(my_config)
```
</pt>
<tf>
Create a model from your custom configuration with [`TFAutoModel.from_config`]:

```py
>>> from transformers import TFAutoModel

>>> my_model = TFAutoModel.from_config(my_config)
```
</tf>
</frameworkcontent>

Take a look at the [Create a custom architecture](./create_a_model) guide for more information about building custom configurations.

## Trainer - a PyTorch optimized training loop

All models are a standard [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) so you can use them in any typical training loop. While you can write your own training loop, 🤗 Transformers provides a [`Trainer`] class for PyTorch, which contains the basic training loop and adds additional functionality for features like distributed training, mixed precision, and more.

Depending on your task, you'll typically pass the following parameters to [`Trainer`]:

1. You'll start with a [`PreTrainedModel`] or a [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module):

   ```py
   >>> from transformers import AutoModelForSequenceClassification

   >>> model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
   ```

2. [`TrainingArguments`] contains the model hyperparameters you can change like learning rate, batch size, and the number of epochs to train for. The default values are used if you don't specify any training arguments:

   ```py
   >>> from transformers import TrainingArguments

   >>> training_args = TrainingArguments(
   ...     output_dir="path/to/save/folder/",
   ...     learning_rate=2e-5,
   ...     per_device_train_batch_size=8,
   ...     per_device_eval_batch_size=8,
   ...     num_train_epochs=2,
   ... )
   ```

3. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:

   ```py
   >>> from transformers import AutoTokenizer

   >>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
   ```

4. Load a dataset:

   ```py
   >>> from datasets import load_dataset

   >>> dataset = load_dataset("rotten_tomatoes")  # doctest: +IGNORE_RESULT
   ```

5. Create a function to tokenize the dataset:

   ```py
   >>> def tokenize_dataset(dataset):
   ...     return tokenizer(dataset["text"])
   ```

   Then apply it over the entire dataset with [`~datasets.Dataset.map`]:

   ```py
   >>> dataset = dataset.map(tokenize_dataset, batched=True)
   ```

6.


key: _node_content
string_value:
{"id_": "d82282e1-6b50-4a00-b469-f031e04d7f50", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "18b1306c-48a5-4818-81d0-d66f73942fa4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c2e137ce0247c07837d893b321b7fc9bbde1aafe5ad0487a8c046d35ef16af0f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fbd00022-f900-4a68-a1fa-1b216d7f4f21", "node_type": "1", "metadata": {}, "hash": "61a8a9c1a5e20ea7b470219acc1c9008b652fbc04245b056eb7b91ddb9f590d6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18599, "end_char_idx": 21737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Depending on your task, you'll typically pass the following parameters to [`Trainer`]:

1. You'll start with a [`PreTrainedModel`] or a [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module):

   ```py
   >>> from transformers import AutoModelForSequenceClassification

   >>> model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
   ```

2. [`TrainingArguments`] contains the model hyperparameters you can change like learning rate, batch size, and the number of epochs to train for. The default values are used if you don't specify any training arguments:

   ```py
   >>> from transformers import TrainingArguments

   >>> training_args = TrainingArguments(
   ...     output_dir="path/to/save/folder/",
   ...     learning_rate=2e-5,
   ...     per_device_train_batch_size=8,
   ...     per_device_eval_batch_size=8,
   ...     num_train_epochs=2,
   ... )
   ```

3. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:

   ```py
   >>> from transformers import AutoTokenizer

   >>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
   ```

4. Load a dataset:

   ```py
   >>> from datasets import load_dataset

   >>> dataset = load_dataset("rotten_tomatoes")  # doctest: +IGNORE_RESULT
   ```

5. Create a function to tokenize the dataset:

   ```py
   >>> def tokenize_dataset(dataset):
   ...     return tokenizer(dataset["text"])
   ```

   Then apply it over the entire dataset with [`~datasets.Dataset.map`]:

   ```py
   >>> dataset = dataset.map(tokenize_dataset, batched=True)
   ```

6. A [`DataCollatorWithPadding`] to create a batch of examples from your dataset:

   ```py
   >>> from transformers import DataCollatorWithPadding

   >>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
   ```

Now gather all these classes in [`Trainer`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=dataset["train"],
...     eval_dataset=dataset["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
... )  # doctest: +SKIP
```

When you're ready, call [`~Trainer.train`] to start training:

```py
>>> trainer.train()  # doctest: +SKIP
```

<Tip>

For tasks - like translation or summarization - that use a sequence-to-sequence model, use the [`Seq2SeqTrainer`] and [`Seq2SeqTrainingArguments`] classes instead.

</Tip>

You can customize the training loop behavior by subclassing the methods inside [`Trainer`]. This allows you to customize features such as the loss function, optimizer, and scheduler. Take a look at the [`Trainer`] reference for which methods can be subclassed. 

The other way to customize the training loop is by using [Callbacks](./main_classes/callback). You can use callbacks to integrate with other libraries and inspect the training loop to report on progress or stop the training early. Callbacks do not modify anything in the training loop itself. To customize something like the loss function, you need to subclass the [`Trainer`] instead.


key: _node_content
string_value:
{"id_": "fbd00022-f900-4a68-a1fa-1b216d7f4f21", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d82282e1-6b50-4a00-b469-f031e04d7f50", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0ea1240657f41850f3ec6410b2930ccf17b54802db76c11b0c45aa349718857e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b69c8a8-950b-425f-84d9-5cfe9f09c543", "node_type": "1", "metadata": {}, "hash": "e44743883676472df595ad995b2d82e1f791f6645c551eaf22ae8065266fee36", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20240, "end_char_idx": 23501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A [`DataCollatorWithPadding`] to create a batch of examples from your dataset:

   ```py
   >>> from transformers import DataCollatorWithPadding

   >>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
   ```

Now gather all these classes in [`Trainer`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=dataset["train"],
...     eval_dataset=dataset["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
... )  # doctest: +SKIP
```

When you're ready, call [`~Trainer.train`] to start training:

```py
>>> trainer.train()  # doctest: +SKIP
```

<Tip>

For tasks - like translation or summarization - that use a sequence-to-sequence model, use the [`Seq2SeqTrainer`] and [`Seq2SeqTrainingArguments`] classes instead.

</Tip>

You can customize the training loop behavior by subclassing the methods inside [`Trainer`]. This allows you to customize features such as the loss function, optimizer, and scheduler. Take a look at the [`Trainer`] reference for which methods can be subclassed. 

The other way to customize the training loop is by using [Callbacks](./main_classes/callback). You can use callbacks to integrate with other libraries and inspect the training loop to report on progress or stop the training early. Callbacks do not modify anything in the training loop itself. To customize something like the loss function, you need to subclass the [`Trainer`] instead.

## Train with TensorFlow

All models are a standard [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model) so they can be trained in TensorFlow with the [Keras](https://keras.io/) API. 🤗 Transformers provides the [`~TFPreTrainedModel.prepare_tf_dataset`] method to easily load your dataset as a `tf.data.Dataset` so you can start training right away with Keras' [`compile`](https://keras.io/api/models/model_training_apis/#compile-method) and [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) methods.

1. You'll start with a [`TFPreTrainedModel`] or a [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model):

   ```py
   >>> from transformers import TFAutoModelForSequenceClassification

   >>> model = TFAutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
   ```

2. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:

   ```py
   >>> from transformers import AutoTokenizer

   >>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
   ```

3. Create a function to tokenize the dataset:

   ```py
   >>> def tokenize_dataset(dataset):
   ...     return tokenizer(dataset["text"])  # doctest: +SKIP
   ```

4. Apply the tokenizer over the entire dataset with [`~datasets.Dataset.map`] and then pass the dataset and tokenizer to [`~TFPreTrainedModel.prepare_tf_dataset`]. You can also change the batch size and shuffle the dataset here if you'd like:

   ```py
   >>> dataset = dataset.map(tokenize_dataset)  # doctest: +SKIP
   >>> tf_dataset = model.prepare_tf_dataset(
   ...     dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer
   ... )  # doctest: +SKIP
   ```

5.


key: _node_content
string_value:
{"id_": "7b69c8a8-950b-425f-84d9-5cfe9f09c543", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fbd00022-f900-4a68-a1fa-1b216d7f4f21", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d0c389fceb632b1a1ac647f273d3735da2868714bf90898663f4be89e8ebea03", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21947, "end_char_idx": 24255, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
🤗 Transformers provides the [`~TFPreTrainedModel.prepare_tf_dataset`] method to easily load your dataset as a `tf.data.Dataset` so you can start training right away with Keras' [`compile`](https://keras.io/api/models/model_training_apis/#compile-method) and [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) methods.

1. You'll start with a [`TFPreTrainedModel`] or a [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model):

   ```py
   >>> from transformers import TFAutoModelForSequenceClassification

   >>> model = TFAutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
   ```

2. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:

   ```py
   >>> from transformers import AutoTokenizer

   >>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
   ```

3. Create a function to tokenize the dataset:

   ```py
   >>> def tokenize_dataset(dataset):
   ...     return tokenizer(dataset["text"])  # doctest: +SKIP
   ```

4. Apply the tokenizer over the entire dataset with [`~datasets.Dataset.map`] and then pass the dataset and tokenizer to [`~TFPreTrainedModel.prepare_tf_dataset`]. You can also change the batch size and shuffle the dataset here if you'd like:

   ```py
   >>> dataset = dataset.map(tokenize_dataset)  # doctest: +SKIP
   >>> tf_dataset = model.prepare_tf_dataset(
   ...     dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer
   ... )  # doctest: +SKIP
   ```

5. When you're ready, you can call `compile` and `fit` to start training. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

   ```py
   >>> from tensorflow.keras.optimizers import Adam

   >>> model.compile(optimizer='adam')  # No loss argument!
   >>> model.fit(tf_dataset)  # doctest: +SKIP
   ```

## What's next?

Now that you've completed the 🤗 Transformers quick tour, check out our guides and learn how to do more specific things like writing a custom model, fine-tuning a model for a task, and how to train a model with a script. If you're interested in learning more about 🤗 Transformers core concepts, grab a cup of coffee and take a look at our Conceptual Guides!


key: _node_content
string_value:
{"id_": "a27b5ed6-5fad-4e93-83d4-d8b119ec3685", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a4950503-fbd5-4367-a378-93fd58e427f8", "node_type": "1", "metadata": {}, "hash": "c73989040327e25a5f79f2fa66909924a0b5781cbb3e570b886f45e06b1107f1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3469, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Trainer

The [`Trainer`] is a complete training and evaluation loop for PyTorch models implemented in the Transformers library. You only need to pass it the necessary pieces for training (model, tokenizer, dataset, evaluation function, training hyperparameters, etc.), and the [`Trainer`] class takes care of the rest. This makes it easier to start training faster without manually writing your own training loop. But at the same time, [`Trainer`] is very customizable and offers a ton of training options so you can tailor it to your exact training needs.

<Tip>

In addition to the [`Trainer`] class, Transformers also provides a [`Seq2SeqTrainer`] class for sequence-to-sequence tasks like translation or summarization. There is also the [`~trl.SFTTrainer`] class from the [TRL](https://hf.co/docs/trl) library which wraps the [`Trainer`] class and is optimized for training language models like Llama-2 and Mistral with autoregressive techniques. [`~trl.SFTTrainer`] also supports features like sequence packing, LoRA, quantization, and DeepSpeed for efficiently scaling to any model size.

<br>

Feel free to check out the [API reference](./main_classes/trainer) for these other [`Trainer`]-type classes to learn more about when to use which one. In general, [`Trainer`] is the most versatile option and is appropriate for a broad spectrum of tasks. [`Seq2SeqTrainer`] is designed for sequence-to-sequence tasks and [`~trl.SFTTrainer`] is designed for training language models.

</Tip>

Before you start, make sure [Accelerate](https://hf.co/docs/accelerate) - a library for enabling and running PyTorch training across distributed environments - is installed.

```bash
pip install accelerate

# upgrade
pip install accelerate --upgrade
```

This guide provides an overview of the [`Trainer`] class.

## Basic usage

[`Trainer`] includes all the code you'll find in a basic training loop:

1. perform a training step to calculate the loss
2. calculate the gradients with the [`~accelerate.Accelerator.backward`] method
3. update the weights based on the gradients
4. repeat this process until you've reached a predetermined number of epochs

The [`Trainer`] class abstracts all of this code away so you don't have to worry about manually writing a training loop every time or if you're just getting started with PyTorch and training. You only need to provide the essential components required for training, such as a model and a dataset, and the [`Trainer`] class handles everything else.

If you want to specify any training options or hyperparameters, you can find them in the [`TrainingArguments`] class. For example, let's define where to save the model in `output_dir` and push the model to the Hub after training with `push_to_hub=True`.

```py
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="your-model",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=2,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    push_to_hub=True,
)
```

Pass `training_args` to the [`Trainer`] along with a model, dataset, something to preprocess the dataset with (depending on your data type it could be a tokenizer, feature extractor or image processor), a data collator, and a function to compute the metrics you want to track during training.

Finally, call [`~Trainer.train`] to start training!


key: _node_content
string_value:
{"id_": "a4950503-fbd5-4367-a378-93fd58e427f8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a27b5ed6-5fad-4e93-83d4-d8b119ec3685", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2ae0b3f3203e112949fbcdf79c05cd37b7c1ba7fc9a8c5d73cae7c1ca9a3f0ef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "50f181fd-0f29-4492-8b32-8f67ad50ab42", "node_type": "1", "metadata": {}, "hash": "b83cef4bfd4a7f7a31fdd599e3d7927cf609105e8ab369dbf3436366f21f9a0a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1808, "end_char_idx": 5274, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Basic usage

[`Trainer`] includes all the code you'll find in a basic training loop:

1. perform a training step to calculate the loss
2. calculate the gradients with the [`~accelerate.Accelerator.backward`] method
3. update the weights based on the gradients
4. repeat this process until you've reached a predetermined number of epochs

The [`Trainer`] class abstracts all of this code away so you don't have to worry about manually writing a training loop every time or if you're just getting started with PyTorch and training. You only need to provide the essential components required for training, such as a model and a dataset, and the [`Trainer`] class handles everything else.

If you want to specify any training options or hyperparameters, you can find them in the [`TrainingArguments`] class. For example, let's define where to save the model in `output_dir` and push the model to the Hub after training with `push_to_hub=True`.

```py
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="your-model",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=2,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    push_to_hub=True,
)
```

Pass `training_args` to the [`Trainer`] along with a model, dataset, something to preprocess the dataset with (depending on your data type it could be a tokenizer, feature extractor or image processor), a data collator, and a function to compute the metrics you want to track during training.

Finally, call [`~Trainer.train`] to start training!

```py
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)

trainer.train()
```

### Checkpoints

The [`Trainer`] class saves your model checkpoints to the directory specified in the `output_dir` parameter of [`TrainingArguments`]. You'll find the checkpoints saved in a `checkpoint-000` subfolder where the numbers at the end correspond to the training step. Saving checkpoints are useful for resuming training later.

```py
# resume from latest checkpoint
trainer.train(resume_from_checkpoint=True)

# resume from specific checkpoint saved in output directory
trainer.train(resume_from_checkpoint="your-model/checkpoint-1000")
```

You can save your checkpoints (the optimizer state is not saved by default) to the Hub by setting `push_to_hub=True` in [`TrainingArguments`] to commit and push them. Other options for deciding how your checkpoints are saved are set up in the [`hub_strategy`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.hub_strategy) parameter:

* `hub_strategy="checkpoint"` pushes the latest checkpoint to a subfolder named "last-checkpoint" from which you can resume training
* `hub_strategy="all_checkpoints"` pushes all checkpoints to the directory defined in `output_dir` (you'll see one checkpoint per folder in your model repository)

When you resume training from a checkpoint, the [`Trainer`] tries to keep the Python, NumPy, and PyTorch RNG states the same as they were when the checkpoint was saved. But because PyTorch has various non-deterministic default settings, the RNG states aren't guaranteed to be the same.


key: _node_content
string_value:
{"id_": "50f181fd-0f29-4492-8b32-8f67ad50ab42", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4950503-fbd5-4367-a378-93fd58e427f8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0e8ee71a435ce365b890aeeff7d763c574b18669bd1b124328866db733a977ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b3e035da-4276-4281-b0e2-1c59f06bfe34", "node_type": "1", "metadata": {}, "hash": "3ac69d12e3599402cd8789085c6f2e23b4d38232fc5618ec5dd5fbd556e8bc2b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3471, "end_char_idx": 7020, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)

trainer.train()
```

### Checkpoints

The [`Trainer`] class saves your model checkpoints to the directory specified in the `output_dir` parameter of [`TrainingArguments`]. You'll find the checkpoints saved in a `checkpoint-000` subfolder where the numbers at the end correspond to the training step. Saving checkpoints are useful for resuming training later.

```py
# resume from latest checkpoint
trainer.train(resume_from_checkpoint=True)

# resume from specific checkpoint saved in output directory
trainer.train(resume_from_checkpoint="your-model/checkpoint-1000")
```

You can save your checkpoints (the optimizer state is not saved by default) to the Hub by setting `push_to_hub=True` in [`TrainingArguments`] to commit and push them. Other options for deciding how your checkpoints are saved are set up in the [`hub_strategy`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.hub_strategy) parameter:

* `hub_strategy="checkpoint"` pushes the latest checkpoint to a subfolder named "last-checkpoint" from which you can resume training
* `hub_strategy="all_checkpoints"` pushes all checkpoints to the directory defined in `output_dir` (you'll see one checkpoint per folder in your model repository)

When you resume training from a checkpoint, the [`Trainer`] tries to keep the Python, NumPy, and PyTorch RNG states the same as they were when the checkpoint was saved. But because PyTorch has various non-deterministic default settings, the RNG states aren't guaranteed to be the same. If you want to enable full determinism, take a look at the [Controlling sources of randomness](https://pytorch.org/docs/stable/notes/randomness#controlling-sources-of-randomness) guide to learn what you can enable to make your training fully deterministic. Keep in mind though that by making certain settings deterministic, training may be slower.

## Customize the Trainer

While the [`Trainer`] class is designed to be accessible and easy-to-use, it also offers a lot of customizability for more adventurous users. Many of the [`Trainer`]'s method can be subclassed and overridden to support the functionality you want, without having to rewrite the entire training loop from scratch to accommodate it. These methods include:

* [`~Trainer.get_train_dataloader`] creates a training DataLoader
* [`~Trainer.get_eval_dataloader`] creates an evaluation DataLoader
* [`~Trainer.get_test_dataloader`] creates a test DataLoader
* [`~Trainer.log`] logs information on the various objects that watch training
* [`~Trainer.create_optimizer_and_scheduler`] creates an optimizer and learning rate scheduler if they weren't passed in the `__init__`; these can also be separately customized with [`~Trainer.create_optimizer`] and [`~Trainer.create_scheduler`] respectively
* [`~Trainer.compute_loss`] computes the loss on a batch of training inputs
* [`~Trainer.training_step`] performs the training step
* [`~Trainer.prediction_step`] performs the prediction and test step
* [`~Trainer.evaluate`] evaluates the model and returns the evaluation metrics
* [`~Trainer.predict`] makes predictions (with metrics if labels are available) on the test set

For example, if you want to customize the [`~Trainer.compute_loss`] method to use a weighted loss instead.


key: _node_content
string_value:
{"id_": "b3e035da-4276-4281-b0e2-1c59f06bfe34", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "50f181fd-0f29-4492-8b32-8f67ad50ab42", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "56a462f16131784ef5b237cef28d3ff0814b24f1860014e2421574792c7e4014", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e196db1-2152-437a-aaf3-21a45fb4c2f5", "node_type": "1", "metadata": {}, "hash": "8096f1b996cc2c648faada935544920a581958d5fb48441d5ee0bb077dd6baaa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5275, "end_char_idx": 8613, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you want to enable full determinism, take a look at the [Controlling sources of randomness](https://pytorch.org/docs/stable/notes/randomness#controlling-sources-of-randomness) guide to learn what you can enable to make your training fully deterministic. Keep in mind though that by making certain settings deterministic, training may be slower.

## Customize the Trainer

While the [`Trainer`] class is designed to be accessible and easy-to-use, it also offers a lot of customizability for more adventurous users. Many of the [`Trainer`]'s method can be subclassed and overridden to support the functionality you want, without having to rewrite the entire training loop from scratch to accommodate it. These methods include:

* [`~Trainer.get_train_dataloader`] creates a training DataLoader
* [`~Trainer.get_eval_dataloader`] creates an evaluation DataLoader
* [`~Trainer.get_test_dataloader`] creates a test DataLoader
* [`~Trainer.log`] logs information on the various objects that watch training
* [`~Trainer.create_optimizer_and_scheduler`] creates an optimizer and learning rate scheduler if they weren't passed in the `__init__`; these can also be separately customized with [`~Trainer.create_optimizer`] and [`~Trainer.create_scheduler`] respectively
* [`~Trainer.compute_loss`] computes the loss on a batch of training inputs
* [`~Trainer.training_step`] performs the training step
* [`~Trainer.prediction_step`] performs the prediction and test step
* [`~Trainer.evaluate`] evaluates the model and returns the evaluation metrics
* [`~Trainer.predict`] makes predictions (with metrics if labels are available) on the test set

For example, if you want to customize the [`~Trainer.compute_loss`] method to use a weighted loss instead.

```py
from torch import nn
from transformers import Trainer

class CustomTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")
        # forward pass
        outputs = model(**inputs)
        logits = outputs.get("logits")
        # compute custom loss for 3 labels with different weights
        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0], device=model.device))
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))
        return (loss, outputs) if return_outputs else loss
```

### Callbacks

Another option for customizing the [`Trainer`] is to use [callbacks](callbacks). Callbacks *don't change* anything in the training loop. They inspect the training loop state and then execute some action (early stopping, logging results, etc.) depending on the state. In other words, a callback can't be used to implement something like a custom loss function and you'll need to subclass and override the [`~Trainer.compute_loss`] method for that.

For example, if you want to add an early stopping callback to the training loop after 10 steps.

```py
from transformers import TrainerCallback

class EarlyStoppingCallback(TrainerCallback):
    def __init__(self, num_steps=10):
        self.num_steps = num_steps
    
    def on_step_end(self, args, state, control, **kwargs):
        if state.global_step >= self.num_steps:
            return {"should_training_stop": True}
        else:
            return {}
```

Then pass it to the [`Trainer`]'s `callback` parameter.


key: _node_content
string_value:
{"id_": "5e196db1-2152-437a-aaf3-21a45fb4c2f5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b3e035da-4276-4281-b0e2-1c59f06bfe34", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "93f42bbec48de48d071215fe083f31fdeae8dc237455dfed1e7bbf2435bc8571", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "18c566d8-5550-4947-acca-0c5d1b431eb4", "node_type": "1", "metadata": {}, "hash": "e4a692305e1d9e04ce8d5de4ed4a2d52c947f5bb02a47025960726f306177531", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7022, "end_char_idx": 10172, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from torch import nn
from transformers import Trainer

class CustomTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")
        # forward pass
        outputs = model(**inputs)
        logits = outputs.get("logits")
        # compute custom loss for 3 labels with different weights
        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0], device=model.device))
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))
        return (loss, outputs) if return_outputs else loss
```

### Callbacks

Another option for customizing the [`Trainer`] is to use [callbacks](callbacks). Callbacks *don't change* anything in the training loop. They inspect the training loop state and then execute some action (early stopping, logging results, etc.) depending on the state. In other words, a callback can't be used to implement something like a custom loss function and you'll need to subclass and override the [`~Trainer.compute_loss`] method for that.

For example, if you want to add an early stopping callback to the training loop after 10 steps.

```py
from transformers import TrainerCallback

class EarlyStoppingCallback(TrainerCallback):
    def __init__(self, num_steps=10):
        self.num_steps = num_steps
    
    def on_step_end(self, args, state, control, **kwargs):
        if state.global_step >= self.num_steps:
            return {"should_training_stop": True}
        else:
            return {}
```

Then pass it to the [`Trainer`]'s `callback` parameter.

```py
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    callback=[EarlyStoppingCallback()],
)
```

## Logging

<Tip>

Check out the [logging](./main_classes/logging) API reference for more information about the different logging levels.

</Tip>

The [`Trainer`] is set to `logging.INFO` by default which reports errors, warnings, and other basic information. A [`Trainer`] replica - in distributed environments - is set to `logging.WARNING` which only reports errors and warnings. You can change the logging level with the [`log_level`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level) and [`log_level_replica`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level_replica) parameters in [`TrainingArguments`].

To configure the log level setting for each node, use the [`log_on_each_node`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.log_on_each_node) parameter to determine whether to use the log level on each node or only on the main node.

<Tip>

[`Trainer`] sets the log level separately for each node in the [`Trainer.__init__`] method, so you may want to consider setting this sooner if you're using other Transformers functionalities before creating the [`Trainer`] object.


key: _node_content
string_value:
{"id_": "18c566d8-5550-4947-acca-0c5d1b431eb4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e196db1-2152-437a-aaf3-21a45fb4c2f5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fa968655c4a983a8d781d6dd507829bf6b8fc6bd95144eb63367ea01de8a11e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e75846be-12af-4f6f-ae78-1833c5a0375b", "node_type": "1", "metadata": {}, "hash": "400af10df9f3ce12b740746bb79063b84391fdd89592d53c2e52ec082b39823b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8615, "end_char_idx": 11854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    callback=[EarlyStoppingCallback()],
)
```

## Logging

<Tip>

Check out the [logging](./main_classes/logging) API reference for more information about the different logging levels.

</Tip>

The [`Trainer`] is set to `logging.INFO` by default which reports errors, warnings, and other basic information. A [`Trainer`] replica - in distributed environments - is set to `logging.WARNING` which only reports errors and warnings. You can change the logging level with the [`log_level`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level) and [`log_level_replica`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level_replica) parameters in [`TrainingArguments`].

To configure the log level setting for each node, use the [`log_on_each_node`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.log_on_each_node) parameter to determine whether to use the log level on each node or only on the main node.

<Tip>

[`Trainer`] sets the log level separately for each node in the [`Trainer.__init__`] method, so you may want to consider setting this sooner if you're using other Transformers functionalities before creating the [`Trainer`] object.

</Tip>

For example, to set your main code and modules to use the same log level according to each node:

```py
logger = logging.getLogger(__name__)

logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
    datefmt="%m/%d/%Y %H:%M:%S",
    handlers=[logging.StreamHandler(sys.stdout)],
)

log_level = training_args.get_process_log_level()
logger.setLevel(log_level)
datasets.utils.logging.set_verbosity(log_level)
transformers.utils.logging.set_verbosity(log_level)

trainer = Trainer(...)
```

Use different combinations of `log_level` and `log_level_replica` to configure what gets logged on each of the nodes.

<hfoptions id="logging">
<hfoption id="single node">

```bash
my_app.py ... --log_level warning --log_level_replica error
```

</hfoption>
<hfoption id="multi-node">

Add the `log_on_each_node 0` parameter for multi-node environments.

```bash
my_app.py ... --log_level warning --log_level_replica error --log_on_each_node 0

# set to only report errors
my_app.py ... --log_level error --log_level_replica error --log_on_each_node 0
```

</hfoption>
</hfoptions>

## NEFTune

[NEFTune](https://hf.co/papers/2310.05914) is a technique that can improve performance by adding noise to the embedding vectors during training. To enable it in [`Trainer`], set the `neftune_noise_alpha` parameter in [`TrainingArguments`] to control how much noise is added.

```py
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(..., neftune_noise_alpha=0.1)
trainer = Trainer(..., args=training_args)
```

NEFTune is disabled after training to restore the original embedding layer to avoid any unexpected behavior.


key: _node_content
string_value:
{"id_": "e75846be-12af-4f6f-ae78-1833c5a0375b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "18c566d8-5550-4947-acca-0c5d1b431eb4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f11fd9635fede0965b253ab910bd1f9e1388073c0a9799d33198a27454f7da5d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "efbbcd84-a4c4-4b3c-9be5-d1b95110a5d0", "node_type": "1", "metadata": {}, "hash": "69d73554b5a1af28f4384900b686b50d8950010d1b591e3f4024b2fabeb756de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10823, "end_char_idx": 12411, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<hfoptions id="logging">
<hfoption id="single node">

```bash
my_app.py ... --log_level warning --log_level_replica error
```

</hfoption>
<hfoption id="multi-node">

Add the `log_on_each_node 0` parameter for multi-node environments.

```bash
my_app.py ... --log_level warning --log_level_replica error --log_on_each_node 0

# set to only report errors
my_app.py ... --log_level error --log_level_replica error --log_on_each_node 0
```

</hfoption>
</hfoptions>

## NEFTune

[NEFTune](https://hf.co/papers/2310.05914) is a technique that can improve performance by adding noise to the embedding vectors during training. To enable it in [`Trainer`], set the `neftune_noise_alpha` parameter in [`TrainingArguments`] to control how much noise is added.

```py
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(..., neftune_noise_alpha=0.1)
trainer = Trainer(..., args=training_args)
```

NEFTune is disabled after training to restore the original embedding layer to avoid any unexpected behavior.

## GaLore

Gradient Low-Rank Projection (GaLore) is a memory-efficient low-rank training strategy that allows full-parameter learning but is more memory-efficient than common low-rank adaptation methods, such as LoRA.

First make sure to install GaLore official repository:

```bash
pip install galore-torch
```

Then simply add one of `["galore_adamw", "galore_adafactor", "galore_adamw_8bit"]` in `optim` together with `optim_target_modules`, which can be a list of strings, regex or full path corresponding to the target module names you want to adapt.


key: _node_content
string_value:
{"id_": "efbbcd84-a4c4-4b3c-9be5-d1b95110a5d0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e75846be-12af-4f6f-ae78-1833c5a0375b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ecf092d95b5960db7b27332137afd2dd9a208a0d763c24de528df0c47b45cb06", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56813570-13c5-4d91-83f3-1ec1376a062e", "node_type": "1", "metadata": {}, "hash": "18d01c6f0a32ae4a4e8cacd52dd342424179017e1952a967df19dc70f5ccb613", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10823, "end_char_idx": 14292, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<hfoptions id="logging">
<hfoption id="single node">

```bash
my_app.py ... --log_level warning --log_level_replica error
```

</hfoption>
<hfoption id="multi-node">

Add the `log_on_each_node 0` parameter for multi-node environments.

```bash
my_app.py ... --log_level warning --log_level_replica error --log_on_each_node 0

# set to only report errors
my_app.py ... --log_level error --log_level_replica error --log_on_each_node 0
```

</hfoption>
</hfoptions>

## NEFTune

[NEFTune](https://hf.co/papers/2310.05914) is a technique that can improve performance by adding noise to the embedding vectors during training. To enable it in [`Trainer`], set the `neftune_noise_alpha` parameter in [`TrainingArguments`] to control how much noise is added.

```py
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(..., neftune_noise_alpha=0.1)
trainer = Trainer(..., args=training_args)
```

NEFTune is disabled after training to restore the original embedding layer to avoid any unexpected behavior.

## GaLore

Gradient Low-Rank Projection (GaLore) is a memory-efficient low-rank training strategy that allows full-parameter learning but is more memory-efficient than common low-rank adaptation methods, such as LoRA.

First make sure to install GaLore official repository:

```bash
pip install galore-torch
```

Then simply add one of `["galore_adamw", "galore_adafactor", "galore_adamw_8bit"]` in `optim` together with `optim_target_modules`, which can be a list of strings, regex or full path corresponding to the target module names you want to adapt. Below is an end-to-end example script (make sure to `pip install trl datasets`):

```python
import torch
import datasets
import trl

from transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-galore",
    max_steps=100,
    per_device_train_batch_size=2,
    optim="galore_adamw",
    optim_target_modules=["attn", "mlp"]
)

model_id = "google/gemma-2b"

config = AutoConfig.from_pretrained(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_config(config).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=512,
)

trainer.train()
```

To pass extra arguments supports by GaLore, you should pass correctly `optim_args`, for example:

```python
import torch
import datasets
import trl

from transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-galore",
    max_steps=100,
    per_device_train_batch_size=2,
    optim="galore_adamw",
    optim_target_modules=["attn", "mlp"],
    optim_args="rank=64, update_proj_gap=100, scale=0.10",
)

model_id = "google/gemma-2b"

config = AutoConfig.from_pretrained(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_config(config).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=512,
)

trainer.train()
```

You can read more about the method in the [original repository](https://github.com/jiaweizzhao/GaLore) or the [paper](https://arxiv.org/abs/2403.03507).


key: _node_content
string_value:
{"id_": "56813570-13c5-4d91-83f3-1ec1376a062e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "efbbcd84-a4c4-4b3c-9be5-d1b95110a5d0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "13375c2d5758885b34051a3feb435f05d5ac8408b583b1146e0878b6f364d5d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "af49548b-3b9a-47b4-99aa-2fa0f52189cc", "node_type": "1", "metadata": {}, "hash": "0e1106b978fc14ae02ea553da24672cc236801000700933f3ba3b4912abeb356", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14294, "end_char_idx": 16849, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Currently you can only train Linear layers that are considered as GaLore layers and will use low-rank decomposition to be trained while remaining layers will be optimized in the conventional manner.

Note it will take a bit of time before starting the training (~3 minutes for a 2B model on a NVIDIA A100), but training should go smoothly afterwards.

You can also perform layer-wise optimization by post-pending the optimizer name with `layerwise` like below:

```python
import torch
import datasets
import trl

from transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-galore",
    max_steps=100,
    per_device_train_batch_size=2,
    optim="galore_adamw_layerwise",
    optim_target_modules=["attn", "mlp"]
)

model_id = "google/gemma-2b"

config = AutoConfig.from_pretrained(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_config(config).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=512,
)

trainer.train()
```

Note layerwise optimization is a bit experimental and does not support DDP (Distributed Data Parallel), thus you can run the training script only on a single GPU. Please see [this appropriate section](https://github.com/jiaweizzhao/GaLore?tab=readme-ov-file#train-7b-model-with-a-single-gpu-with-24gb-memory) for more details. Other features such as gradient clipping, DeepSpeed, etc might not be supported out of the box. Please [raise an issue on GitHub](https://github.com/huggingface/transformers/issues) if you encounter such issue.

## LOMO optimizer

The LOMO optimizers have been introduced in [Full Parameter Fine-Tuning for Large Language Models with Limited Resources](https://hf.co/papers/2306.09782) and [AdaLomo: Low-memory Optimization with Adaptive Learning Rate](https://hf.co/papers/2310.10195). 
They both consist of an efficient full-parameter fine-tuning method. These optimizers fuse the gradient computation and the parameter update in one step to reduce memory usage. Supported optimizers for LOMO are `"lomo"` and `"adalomo"`. First either install LOMO from pypi `pip install lomo-optim` or install it from source with `pip install git+https://github.com/OpenLMLab/LOMO.git`. 

<Tip>

According to the authors, it is recommended to use `AdaLomo` without `grad_norm` to get better performance and higher throughput.


key: _node_content
string_value:
{"id_": "af49548b-3b9a-47b4-99aa-2fa0f52189cc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "56813570-13c5-4d91-83f3-1ec1376a062e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5c3b7d436a06b7e256d324c1b547dbd12e359e715a1af59003f31c9ea93e288b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b964ab35-75c8-452a-8124-c4e3a4e2b96b", "node_type": "1", "metadata": {}, "hash": "4d26d021b0bb1ef0193d0213ecd443fdfa444ffc43406d77aa57c29dad3060cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15673, "end_char_idx": 18667, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Please see [this appropriate section](https://github.com/jiaweizzhao/GaLore?tab=readme-ov-file#train-7b-model-with-a-single-gpu-with-24gb-memory) for more details. Other features such as gradient clipping, DeepSpeed, etc might not be supported out of the box. Please [raise an issue on GitHub](https://github.com/huggingface/transformers/issues) if you encounter such issue.

## LOMO optimizer

The LOMO optimizers have been introduced in [Full Parameter Fine-Tuning for Large Language Models with Limited Resources](https://hf.co/papers/2306.09782) and [AdaLomo: Low-memory Optimization with Adaptive Learning Rate](https://hf.co/papers/2310.10195). 
They both consist of an efficient full-parameter fine-tuning method. These optimizers fuse the gradient computation and the parameter update in one step to reduce memory usage. Supported optimizers for LOMO are `"lomo"` and `"adalomo"`. First either install LOMO from pypi `pip install lomo-optim` or install it from source with `pip install git+https://github.com/OpenLMLab/LOMO.git`. 

<Tip>

According to the authors, it is recommended to use `AdaLomo` without `grad_norm` to get better performance and higher throughput.

</Tip>

Below is a simple script to demonstrate how to fine-tune [google/gemma-2b](https://huggingface.co/google/gemma-2b) on IMDB dataset in full precision:

```python
import torch
import datasets
from transformers import TrainingArguments, AutoTokenizer, AutoModelForCausalLM
import trl

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-lomo",
    max_steps=1000,
    per_device_train_batch_size=4,
    optim="adalomo",
    gradient_checkpointing=True,
    logging_strategy="steps",
    logging_steps=1,
    learning_rate=2e-6,
    save_strategy="no",
    run_name="lomo-imdb",
)

model_id = "google/gemma-2b"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, low_cpu_mem_usage=True).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=1024,
)

trainer.train()
```

## Accelerate and Trainer

The [`Trainer`] class is powered by [Accelerate](https://hf.co/docs/accelerate), a library for easily training PyTorch models in distributed environments with support for integrations such as [FullyShardedDataParallel (FSDP)](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/) and [DeepSpeed](https://www.deepspeed.ai/).

<Tip>

Learn more about FSDP sharding strategies, CPU offloading, and more with the [`Trainer`] in the [Fully Sharded Data Parallel](fsdp) guide.

</Tip>

To use Accelerate with [`Trainer`], run the [`accelerate.config`](https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-config) command to set up training for your training environment. This command creates a `config_file.yaml` that'll be used when you launch your training script.


key: _node_content
string_value:
{"id_": "b964ab35-75c8-452a-8124-c4e3a4e2b96b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "af49548b-3b9a-47b4-99aa-2fa0f52189cc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ba946ff824f0fdf5ede7d02e83a44c5e04113f480a259be1512365ddca367609", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e94485e2-d25e-454d-ac88-5129c0359c45", "node_type": "1", "metadata": {}, "hash": "b86580875dbfe86cc0625fe7769988277b2a76725355a704803eb4d60bf484cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18213, "end_char_idx": 21542, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

Learn more about FSDP sharding strategies, CPU offloading, and more with the [`Trainer`] in the [Fully Sharded Data Parallel](fsdp) guide.

</Tip>

To use Accelerate with [`Trainer`], run the [`accelerate.config`](https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-config) command to set up training for your training environment. This command creates a `config_file.yaml` that'll be used when you launch your training script. For example, some example configurations you can setup are:

<hfoptions id="config">
<hfoption id="DistributedDataParallel">

```yml
compute_environment: LOCAL_MACHINE                                                                                             
distributed_type: MULTI_GPU                                                                                                    
downcast_bf16: 'no'
gpu_ids: all
machine_rank: 0 #change rank as per the node
main_process_ip: 192.168.20.1
main_process_port: 9898
main_training_function: main
mixed_precision: fp16
num_machines: 2
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

</hfoption>
<hfoption id="FSDP">

```yml
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_forward_prefetch: true
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_transformer_layer_cls_to_wrap: BertLayer
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

</hfoption>
<hfoption id="DeepSpeed">

```yml
compute_environment: LOCAL_MACHINE
deepspeed_config:
  deepspeed_config_file: /home/user/configs/ds_zero3_config.json
  zero3_init_flag: true
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

</hfoption>
<hfoption id="DeepSpeed with Accelerate plugin">

```yml
compute_environment: LOCAL_MACHINE                                                                                             
deepspeed_config:                                                                                                              
  gradient_accumulation_steps: 1
  gradient_clipping: 0.7
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: true
  zero_stage: 2
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

</hfoption>
</hfoptions>

The [`accelerate_launch`](https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-launch) command is the recommended way to launch your training script on a distributed system with Accelerate and [`Trainer`] with the parameters specified in `config_file.yaml`.


key: _node_content
string_value:
{"id_": "e94485e2-d25e-454d-ac88-5129c0359c45", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b964ab35-75c8-452a-8124-c4e3a4e2b96b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "486b2d28e76b6c93b8283ea374a5d5ffb5baacc6be23a6f4588883357b58247b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21543, "end_char_idx": 23199, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This file is saved to the Accelerate cache folder and automatically loaded when you run `accelerate_launch`.

For example, to run the [run_glue.py](https://github.com/huggingface/transformers/blob/f4db565b695582891e43a5e042e5d318e28f20b8/examples/pytorch/text-classification/run_glue.py#L4) training script with the FSDP configuration:

```bash
accelerate launch \
    ./examples/pytorch/text-classification/run_glue.py \
    --model_name_or_path google-bert/bert-base-cased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --max_seq_length 128 \
    --per_device_train_batch_size 16 \
    --learning_rate 5e-5 \
    --num_train_epochs 3 \
    --output_dir /tmp/$TASK_NAME/ \
    --overwrite_output_dir
```

You could also specify the parameters from the `config_file.yaml` file directly in the command line:

```bash
accelerate launch --num_processes=2 \
    --use_fsdp \
    --mixed_precision=bf16 \
    --fsdp_auto_wrap_policy=TRANSFORMER_BASED_WRAP  \
    --fsdp_transformer_layer_cls_to_wrap="BertLayer" \
    --fsdp_sharding_strategy=1 \
    --fsdp_state_dict_type=FULL_STATE_DICT \
    ./examples/pytorch/text-classification/run_glue.py
    --model_name_or_path google-bert/bert-base-cased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --max_seq_length 128 \
    --per_device_train_batch_size 16 \
    --learning_rate 5e-5 \
    --num_train_epochs 3 \
    --output_dir /tmp/$TASK_NAME/ \
    --overwrite_output_dir
```

Check out the [Launching your Accelerate scripts](https://huggingface.co/docs/accelerate/basic_tutorials/launch) tutorial to learn more about `accelerate_launch` and custom configurations.


key: _node_content
string_value:
{"id_": "1b16d9bc-c728-4491-9319-20c3321b7b01", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4be9ba97-0e37-4ebe-8a16-09837e23a1a4", "node_type": "1", "metadata": {}, "hash": "69d81e0588f6a06e36c9b5d8e4f960e87c81dbbb4bee32daa3509f7272d6abc0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3004, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Train with a script

Along with the 🤗 Transformers [notebooks](./notebooks), there are also example scripts demonstrating how to train a model for a task with [PyTorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch), [TensorFlow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow), or [JAX/Flax](https://github.com/huggingface/transformers/tree/main/examples/flax).

You will also find scripts we've used in our [research projects](https://github.com/huggingface/transformers/tree/main/examples/research_projects) and [legacy examples](https://github.com/huggingface/transformers/tree/main/examples/legacy) which are mostly community contributed. These scripts are not actively maintained and require a specific version of 🤗 Transformers that will most likely be incompatible with the latest version of the library.

The example scripts are not expected to work out-of-the-box on every problem, and you may need to adapt the script to the problem you're trying to solve. To help you with this, most of the scripts fully expose how data is preprocessed, allowing you to edit it as necessary for your use case.

For any feature you'd like to implement in an example script, please discuss it on the [forum](https://discuss.huggingface.co/) or in an [issue](https://github.com/huggingface/transformers/issues) before submitting a Pull Request. While we welcome bug fixes, it is unlikely we will merge a Pull Request that adds more functionality at the cost of readability.

This guide will show you how to run an example summarization training script in [PyTorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [TensorFlow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization). All examples are expected to work with both frameworks unless otherwise specified.

## Setup

To successfully run the latest version of the example scripts, you have to **install 🤗 Transformers from source** in a new virtual environment:

```bash
git clone https://github.com/huggingface/transformers
cd transformers
pip install .
```

For older versions of the example scripts, click on the toggle below:

<details>
  <summary>Examples for older versions of 🤗 Transformers</summary>
	<ul>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.5.1/examples">v4.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.4.2/examples">v4.4.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.3.3/examples">v4.3.3</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.2.2/examples">v4.2.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.1.1/examples">v4.1.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.0.1/examples">v4.0.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.5.1/examples">v3.5.1</a></li>
		<li><a href="https://github.


key: _node_content
string_value:
{"id_": "4be9ba97-0e37-4ebe-8a16-09837e23a1a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1b16d9bc-c728-4491-9319-20c3321b7b01", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "81194525bbd140e949ff731ea8465f3acb9b9e6645196a34325ce3463207f5a4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "691b5b28-0e79-490f-a5b2-c0c0e42edab4", "node_type": "1", "metadata": {}, "hash": "458b5668fdbef749238e11a3f8337f43be0500b308455f510069d1633a212001", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1805, "end_char_idx": 4034, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All examples are expected to work with both frameworks unless otherwise specified.

## Setup

To successfully run the latest version of the example scripts, you have to **install 🤗 Transformers from source** in a new virtual environment:

```bash
git clone https://github.com/huggingface/transformers
cd transformers
pip install .
```

For older versions of the example scripts, click on the toggle below:

<details>
  <summary>Examples for older versions of 🤗 Transformers</summary>
	<ul>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.5.1/examples">v4.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.4.2/examples">v4.4.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.3.3/examples">v4.3.3</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.2.2/examples">v4.2.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.1.1/examples">v4.1.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.0.1/examples">v4.0.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.5.1/examples">v3.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.4.0/examples">v3.4.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.3.1/examples">v3.3.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.2.0/examples">v3.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.1.0/examples">v3.1.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.0.2/examples">v3.0.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.11.0/examples">v2.11.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.10.0/examples">v2.10.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.9.1/examples">v2.9.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.8.0/examples">v2.8.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.7.0/examples">v2.7.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.6.0/examples">v2.6.


key: _node_content
string_value:
{"id_": "691b5b28-0e79-490f-a5b2-c0c0e42edab4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4be9ba97-0e37-4ebe-8a16-09837e23a1a4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fe46d89f61a70d145390246ad7ff0342190635d32555166271726a8a4dfd72ec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "967628fb-ac4c-4955-b87b-7c01bebffacf", "node_type": "1", "metadata": {}, "hash": "a675129f05031861d2a77a33bee7e5eb4300fa8c381eeee7ca4c27744c7713ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3041, "end_char_idx": 5066, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4.0/examples">v3.4.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.3.1/examples">v3.3.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.2.0/examples">v3.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.1.0/examples">v3.1.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.0.2/examples">v3.0.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.11.0/examples">v2.11.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.10.0/examples">v2.10.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.9.1/examples">v2.9.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.8.0/examples">v2.8.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.7.0/examples">v2.7.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.6.0/examples">v2.6.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.5.1/examples">v2.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.4.0/examples">v2.4.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.3.0/examples">v2.3.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.2.0/examples">v2.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.1.0/examples">v2.1.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.0.0/examples">v2.0.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.2.0/examples">v1.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.1.0/examples">v1.1.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.0.0/examples">v1.0.0</a></li>
	</ul>
</details>

Then switch your current clone of 🤗 Transformers to a specific version, like v3.5.1 for example:

```bash
git checkout tags/v3.5.


key: _node_content
string_value:
{"id_": "967628fb-ac4c-4955-b87b-7c01bebffacf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "691b5b28-0e79-490f-a5b2-c0c0e42edab4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "676fa360e1a3caac595995afe7fb2546f91deb376e2fe03e02271b2d6366a1eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25f64b7e-6d6a-455c-8a99-8a2c6229fd02", "node_type": "1", "metadata": {}, "hash": "b4e9aefd194a9462fde45f20ed8dcc2f3017d3c3280fea6c45b2194f1bd8fa91", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4017, "end_char_idx": 6577, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0/examples">v2.6.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.5.1/examples">v2.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.4.0/examples">v2.4.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.3.0/examples">v2.3.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.2.0/examples">v2.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.1.0/examples">v2.1.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.0.0/examples">v2.0.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.2.0/examples">v1.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.1.0/examples">v1.1.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v1.0.0/examples">v1.0.0</a></li>
	</ul>
</details>

Then switch your current clone of 🤗 Transformers to a specific version, like v3.5.1 for example:

```bash
git checkout tags/v3.5.1
```

After you've setup the correct library version, navigate to the example folder of your choice and install the example specific requirements:

```bash
pip install -r requirements.txt
```

## Run a script

<frameworkcontent>
<pt>
The example script downloads and preprocesses a dataset from the 🤗 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset with the [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```
</pt>
<tf>
The example script downloads and preprocesses a dataset from the 🤗 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset using Keras on an architecture that supports summarization.


key: _node_content
string_value:
{"id_": "25f64b7e-6d6a-455c-8a99-8a2c6229fd02", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "967628fb-ac4c-4955-b87b-7c01bebffacf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1834ed163e104c00e2d17d2797e0a5d91efe9315a30d534aad111edd35d6180d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b306508-aa63-45ac-9be6-1ae3bc33cb09", "node_type": "1", "metadata": {}, "hash": "73402c8f930f349a1f30656c4a78f8fda594359a4fbe07a47e5cc1e6a12b55eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5019, "end_char_idx": 7666, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1 for example:

```bash
git checkout tags/v3.5.1
```

After you've setup the correct library version, navigate to the example folder of your choice and install the example specific requirements:

```bash
pip install -r requirements.txt
```

## Run a script

<frameworkcontent>
<pt>
The example script downloads and preprocesses a dataset from the 🤗 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset with the [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```
</pt>
<tf>
The example script downloads and preprocesses a dataset from the 🤗 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset using Keras on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.

```bash
python examples/tensorflow/summarization/run_summarization.py  \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval
```
</tf>
</frameworkcontent>

## Distributed training and mixed precision

The [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) supports distributed training and mixed precision, which means you can also use it in a script. To enable both of these features:

- Add the `fp16` argument to enable mixed precision.
- Set the number of GPUs to use with the `nproc_per_node` argument.


key: _node_content
string_value:
{"id_": "7b306508-aa63-45ac-9be6-1ae3bc33cb09", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "25f64b7e-6d6a-455c-8a99-8a2c6229fd02", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3c3539dbb75a87e07eda29df9b1f1db917885def146d6bb3e4ee2e69efa24038", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f04a6330-8086-44d2-aeca-515ed13ec2db", "node_type": "1", "metadata": {}, "hash": "db016caf71e3c5138db8a105c02f04fe9b299a401e03dfc07de1d80fb4bb9f10", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6481, "end_char_idx": 9409, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then the script fine-tunes a dataset using Keras on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.

```bash
python examples/tensorflow/summarization/run_summarization.py  \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval
```
</tf>
</frameworkcontent>

## Distributed training and mixed precision

The [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) supports distributed training and mixed precision, which means you can also use it in a script. To enable both of these features:

- Add the `fp16` argument to enable mixed precision.
- Set the number of GPUs to use with the `nproc_per_node` argument.

```bash
torchrun \
    --nproc_per_node 8 pytorch/summarization/run_summarization.py \
    --fp16 \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```

TensorFlow scripts utilize a [`MirroredStrategy`](https://www.tensorflow.org/guide/distributed_training#mirroredstrategy) for distributed training, and you don't need to add any additional arguments to the training script. The TensorFlow script will use multiple GPUs by default if they are available.

## Run a script on a TPU

<frameworkcontent>
<pt>
Tensor Processing Units (TPUs) are specifically designed to accelerate performance. PyTorch supports TPUs with the [XLA](https://www.tensorflow.org/xla) deep learning compiler (see [here](https://github.com/pytorch/xla/blob/master/README.md) for more details). To use a TPU, launch the `xla_spawn.py` script and use the `num_cores` argument to set the number of TPU cores you want to use.

```bash
python xla_spawn.py --num_cores 8 \
    summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```
</pt>
<tf>
Tensor Processing Units (TPUs) are specifically designed to accelerate performance.


key: _node_content
string_value:
{"id_": "f04a6330-8086-44d2-aeca-515ed13ec2db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b306508-aa63-45ac-9be6-1ae3bc33cb09", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "98cc8dcd27d455ec4c822dcc0f605c9c77add1984142c01c794a4c9669d5903e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0358a28-7c7f-454d-9939-d5def0f9d6d6", "node_type": "1", "metadata": {}, "hash": "e054c4c38fb87a613a063b5b3a907454323311f6c90d085eda22374dcc27a79c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8351, "end_char_idx": 10750, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The TensorFlow script will use multiple GPUs by default if they are available.

## Run a script on a TPU

<frameworkcontent>
<pt>
Tensor Processing Units (TPUs) are specifically designed to accelerate performance. PyTorch supports TPUs with the [XLA](https://www.tensorflow.org/xla) deep learning compiler (see [here](https://github.com/pytorch/xla/blob/master/README.md) for more details). To use a TPU, launch the `xla_spawn.py` script and use the `num_cores` argument to set the number of TPU cores you want to use.

```bash
python xla_spawn.py --num_cores 8 \
    summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```
</pt>
<tf>
Tensor Processing Units (TPUs) are specifically designed to accelerate performance. TensorFlow scripts utilize a [`TPUStrategy`](https://www.tensorflow.org/guide/distributed_training#tpustrategy) for training on TPUs. To use a TPU, pass the name of the TPU resource to the `tpu` argument.

```bash
python run_summarization.py  \
    --tpu name_of_tpu_resource \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval
```
</tf>
</frameworkcontent>

## Run a script with 🤗 Accelerate

🤗 [Accelerate](https://huggingface.co/docs/accelerate) is a PyTorch-only library that offers a unified method for training a model on several types of setups (CPU-only, multiple GPUs, TPUs) while maintaining complete visibility into the PyTorch training loop. Make sure you have 🤗 Accelerate installed if you don't already have it:

> Note: As Accelerate is rapidly developing, the git version of accelerate must be installed to run the scripts
```bash
pip install git+https://github.com/huggingface/accelerate
```

Instead of the `run_summarization.py` script, you need to use the `run_summarization_no_trainer.py` script. 🤗 Accelerate supported scripts will have a `task_no_trainer.py` file in the folder.


key: _node_content
string_value:
{"id_": "f0358a28-7c7f-454d-9939-d5def0f9d6d6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f04a6330-8086-44d2-aeca-515ed13ec2db", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a8462377349e20b8ad7b0533073cffd737b7b3e0cb810ddbc83297ee6b481244", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c0b858fe-f154-45b3-8c88-f595ef471807", "node_type": "1", "metadata": {}, "hash": "0c2ce3951a5b8f78bb76ac0d2b436d4947b0aaf83957d6d38a05272713026635", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9410, "end_char_idx": 12467, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
TensorFlow scripts utilize a [`TPUStrategy`](https://www.tensorflow.org/guide/distributed_training#tpustrategy) for training on TPUs. To use a TPU, pass the name of the TPU resource to the `tpu` argument.

```bash
python run_summarization.py  \
    --tpu name_of_tpu_resource \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval
```
</tf>
</frameworkcontent>

## Run a script with 🤗 Accelerate

🤗 [Accelerate](https://huggingface.co/docs/accelerate) is a PyTorch-only library that offers a unified method for training a model on several types of setups (CPU-only, multiple GPUs, TPUs) while maintaining complete visibility into the PyTorch training loop. Make sure you have 🤗 Accelerate installed if you don't already have it:

> Note: As Accelerate is rapidly developing, the git version of accelerate must be installed to run the scripts
```bash
pip install git+https://github.com/huggingface/accelerate
```

Instead of the `run_summarization.py` script, you need to use the `run_summarization_no_trainer.py` script. 🤗 Accelerate supported scripts will have a `task_no_trainer.py` file in the folder. Begin by running the following command to create and save a configuration file:

```bash
accelerate config
```

Test your setup to make sure it is configured correctly:

```bash
accelerate test
```

Now you are ready to launch the training:

```bash
accelerate launch run_summarization_no_trainer.py \
    --model_name_or_path google-t5/t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir ~/tmp/tst-summarization
```

## Use a custom dataset

The summarization script supports custom datasets as long as they are a CSV or JSON Line file. When you use your own dataset, you need to specify several additional arguments:

- `train_file` and `validation_file` specify the path to your training and validation files.
- `text_column` is the input text to summarize.
- `summary_column` is the target text to output.

A summarization script using a custom dataset would look like this:

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --train_file path_to_csv_or_jsonlines_file \
    --validation_file path_to_csv_or_jsonlines_file \
    --text_column text_column_name \
    --summary_column summary_column_name \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --overwrite_output_dir \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --predict_with_generate
```

## Test a script

It is often a good idea to run your script on a smaller number of dataset examples to ensure everything works as expected before committing to an entire dataset which may take hours to complete.


key: _node_content
string_value:
{"id_": "c0b858fe-f154-45b3-8c88-f595ef471807", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0358a28-7c7f-454d-9939-d5def0f9d6d6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9c495a79934e6b4cd7e742a8eae4233ab92e62c506a0be0db76e6da964146957", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "952cd0a2-b08f-448a-8cf2-fb5fca6455c1", "node_type": "1", "metadata": {}, "hash": "785d9ac72f06f758f47f399e8d1fd78d291a871f9cbff4d221e7a83bc34cae0b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11367, "end_char_idx": 14432, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When you use your own dataset, you need to specify several additional arguments:

- `train_file` and `validation_file` specify the path to your training and validation files.
- `text_column` is the input text to summarize.
- `summary_column` is the target text to output.

A summarization script using a custom dataset would look like this:

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --train_file path_to_csv_or_jsonlines_file \
    --validation_file path_to_csv_or_jsonlines_file \
    --text_column text_column_name \
    --summary_column summary_column_name \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --overwrite_output_dir \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --predict_with_generate
```

## Test a script

It is often a good idea to run your script on a smaller number of dataset examples to ensure everything works as expected before committing to an entire dataset which may take hours to complete. Use the following arguments to truncate the dataset to a maximum number of samples:

- `max_train_samples`
- `max_eval_samples`
- `max_predict_samples`

```bash
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google-t5/t5-small \
    --max_train_samples 50 \
    --max_eval_samples 50 \
    --max_predict_samples 50 \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```

Not all example scripts support the `max_predict_samples` argument. If you aren't sure whether your script supports this argument, add the `-h` argument to check:

```bash
examples/pytorch/summarization/run_summarization.py -h
```

## Resume training from checkpoint

Another helpful option to enable is resuming training from a previous checkpoint. This will ensure you can pick up where you left off without starting over if your training gets interrupted. There are two methods to resume training from a checkpoint.

The first method uses the `output_dir previous_output_dir` argument to resume training from the latest checkpoint stored in `output_dir`. In this case, you should remove `overwrite_output_dir`:

```bash
python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --output_dir previous_output_dir \
    --predict_with_generate
```

The second method uses the `resume_from_checkpoint path_to_specific_checkpoint` argument to resume training from a specific checkpoint folder.


key: _node_content
string_value:
{"id_": "952cd0a2-b08f-448a-8cf2-fb5fca6455c1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c0b858fe-f154-45b3-8c88-f595ef471807", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e5546bef06f943394af4b7891ee53a15ba8253b76638bcfd867251278cbf464b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13206, "end_char_idx": 16043, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you aren't sure whether your script supports this argument, add the `-h` argument to check:

```bash
examples/pytorch/summarization/run_summarization.py -h
```

## Resume training from checkpoint

Another helpful option to enable is resuming training from a previous checkpoint. This will ensure you can pick up where you left off without starting over if your training gets interrupted. There are two methods to resume training from a checkpoint.

The first method uses the `output_dir previous_output_dir` argument to resume training from the latest checkpoint stored in `output_dir`. In this case, you should remove `overwrite_output_dir`:

```bash
python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --output_dir previous_output_dir \
    --predict_with_generate
```

The second method uses the `resume_from_checkpoint path_to_specific_checkpoint` argument to resume training from a specific checkpoint folder.

```bash
python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --resume_from_checkpoint path_to_specific_checkpoint \
    --predict_with_generate
```

## Share your model

All scripts can upload your final model to the [Model Hub](https://huggingface.co/models). Make sure you are logged into Hugging Face before you begin:

```bash
huggingface-cli login
```

Then add the `push_to_hub` argument to the script. This argument will create a repository with your Hugging Face username and the folder name specified in `output_dir`.

To give your repository a specific name, use the `push_to_hub_model_id` argument to add it. The repository will be automatically listed under your namespace.

The following example shows how to upload a model with a specific repository name:

```bash
python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path google-t5/t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --push_to_hub \
    --push_to_hub_model_id finetuned-t5-cnn_dailymail \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```


key: _node_content
string_value:
{"id_": "62419187-839c-4569-a228-3457c3cf3a7f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9eccb4f8-9be0-4b4f-b885-0425297592c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f2b284f6ad534dc2c03c12a7a88c3d4b7d0570b5d8b7a08892d964779e6d1930", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3842b001-5910-4ece-a524-776da1372b89", "node_type": "1", "metadata": {}, "hash": "1b334b64c753c618cac8b4d6a5560427e8d1cdac3887a60e1c9a8b833de2d554", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Building custom models

The 🤗 Transformers library is designed to be easily extensible. Every model is fully coded in a given subfolder
of the repository with no abstraction, so you can easily copy a modeling file and tweak it to your needs.

If you are writing a brand new model, it might be easier to start from scratch. In this tutorial, we will show you
how to write a custom model and its configuration so it can be used inside Transformers, and how you can share it
with the community (with the code it relies on) so that anyone can use it, even if it's not present in the 🤗
Transformers library. We'll see how to build upon transformers and extend the framework with your hooks and
custom code.

We will illustrate all of this on a ResNet model, by wrapping the ResNet class of the
[timm library](https://github.com/rwightman/pytorch-image-models) into a [`PreTrainedModel`].

## Writing a custom configuration

Before we dive into the model, let's first write its configuration. The configuration of a model is an object that
will contain all the necessary information to build the model. As we will see in the next section, the model can only
take a `config` to be initialized, so we really need that object to be as complete as possible.

<Tip>

Models in the `transformers` library itself generally follow the convention that they accept a `config` object
in their `__init__` method, and then pass the whole `config` to sub-layers in the model, rather than breaking the 
config object into multiple arguments that are all passed individually to sub-layers. Writing your model in this 
style results in simpler code with a clear "source of truth" for any hyperparameters, and also makes it easier
to reuse code from other models in `transformers`.

</Tip>

In our example, we will take a couple of arguments of the ResNet class that we might want to tweak. Different
configurations will then give us the different types of ResNets that are possible. We then just store those arguments,
after checking the validity of a few of them.

```python
from transformers import PretrainedConfig
from typing import List


class ResnetConfig(PretrainedConfig):
    model_type = "resnet"

    def __init__(
        self,
        block_type="bottleneck",
        layers: List[int] = [3, 4, 6, 3],
        num_classes: int = 1000,
        input_channels: int = 3,
        cardinality: int = 1,
        base_width: int = 64,
        stem_width: int = 64,
        stem_type: str = "",
        avg_down: bool = False,
        **kwargs,
    ):
        if block_type not in ["basic", "bottleneck"]:
            raise ValueError(f"`block_type` must be 'basic' or bottleneck', got {block_type}.")
        if stem_type not in ["", "deep", "deep-tiered"]:
            raise ValueError(f"`stem_type` must be '', 'deep' or 'deep-tiered', got {stem_type}.")


key: _node_content
string_value:
{"id_": "3842b001-5910-4ece-a524-776da1372b89", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9eccb4f8-9be0-4b4f-b885-0425297592c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f2b284f6ad534dc2c03c12a7a88c3d4b7d0570b5d8b7a08892d964779e6d1930", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "62419187-839c-4569-a228-3457c3cf3a7f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1b6a563a111536de11ba91960c6fe56bb8a4d85b43556c1aa7892d9185be916d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "89d70d17-62dc-43e0-9dad-53859d15bfde", "node_type": "1", "metadata": {}, "hash": "4e25843629c9596925a099d680f7bd20bdcdb649c5dedad0acdab0788f43559e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2123, "end_char_idx": 5270, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class ResnetConfig(PretrainedConfig):
    model_type = "resnet"

    def __init__(
        self,
        block_type="bottleneck",
        layers: List[int] = [3, 4, 6, 3],
        num_classes: int = 1000,
        input_channels: int = 3,
        cardinality: int = 1,
        base_width: int = 64,
        stem_width: int = 64,
        stem_type: str = "",
        avg_down: bool = False,
        **kwargs,
    ):
        if block_type not in ["basic", "bottleneck"]:
            raise ValueError(f"`block_type` must be 'basic' or bottleneck', got {block_type}.")
        if stem_type not in ["", "deep", "deep-tiered"]:
            raise ValueError(f"`stem_type` must be '', 'deep' or 'deep-tiered', got {stem_type}.")

        self.block_type = block_type
        self.layers = layers
        self.num_classes = num_classes
        self.input_channels = input_channels
        self.cardinality = cardinality
        self.base_width = base_width
        self.stem_width = stem_width
        self.stem_type = stem_type
        self.avg_down = avg_down
        super().__init__(**kwargs)
```

The three important things to remember when writing you own configuration are the following:
- you have to inherit from `PretrainedConfig`,
- the `__init__` of your `PretrainedConfig` must accept any kwargs,
- those `kwargs` need to be passed to the superclass `__init__`.

The inheritance is to make sure you get all the functionality from the 🤗 Transformers library, while the two other
constraints come from the fact a `PretrainedConfig` has more fields than the ones you are setting. When reloading a
config with the `from_pretrained` method, those fields need to be accepted by your config and then sent to the
superclass.

Defining a `model_type` for your configuration (here `model_type="resnet"`) is not mandatory, unless you want to
register your model with the auto classes (see last section).

With this done, you can easily create and save your configuration like you would do with any other model config of the
library. Here is how we can create a resnet50d config and save it:

```py
resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d_config.save_pretrained("custom-resnet")
```

This will save a file named `config.json` inside the folder `custom-resnet`. You can then reload your config with the
`from_pretrained` method:

```py
resnet50d_config = ResnetConfig.from_pretrained("custom-resnet")
```

You can also use any other method of the [`PretrainedConfig`] class, like [`~PretrainedConfig.push_to_hub`] to
directly upload your config to the Hub.

## Writing a custom model

Now that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that
extracts the hidden features from a batch of images (like [`BertModel`]) and one that is suitable for image
classification (like [`BertForSequenceClassification`]).

As we mentioned before, we'll only write a loose wrapper of the model to keep it simple for this example. The only
thing we need to do before writing this class is a map between the block types and actual block classes.


key: _node_content
string_value:
{"id_": "89d70d17-62dc-43e0-9dad-53859d15bfde", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9eccb4f8-9be0-4b4f-b885-0425297592c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f2b284f6ad534dc2c03c12a7a88c3d4b7d0570b5d8b7a08892d964779e6d1930", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3842b001-5910-4ece-a524-776da1372b89", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "564b16d62c99f70db91d2c3a78ee50e94d53e0736062b65ecbffd087afcd7e25", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "672d03d0-b1e5-4cad-9707-8223405a0550", "node_type": "1", "metadata": {}, "hash": "224883b5d9e07885d738dd4d88feaafee7fa0b56166cd2dbc455738e64695e2e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3703, "end_char_idx": 6347, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When reloading a
config with the `from_pretrained` method, those fields need to be accepted by your config and then sent to the
superclass.

Defining a `model_type` for your configuration (here `model_type="resnet"`) is not mandatory, unless you want to
register your model with the auto classes (see last section).

With this done, you can easily create and save your configuration like you would do with any other model config of the
library. Here is how we can create a resnet50d config and save it:

```py
resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d_config.save_pretrained("custom-resnet")
```

This will save a file named `config.json` inside the folder `custom-resnet`. You can then reload your config with the
`from_pretrained` method:

```py
resnet50d_config = ResnetConfig.from_pretrained("custom-resnet")
```

You can also use any other method of the [`PretrainedConfig`] class, like [`~PretrainedConfig.push_to_hub`] to
directly upload your config to the Hub.

## Writing a custom model

Now that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that
extracts the hidden features from a batch of images (like [`BertModel`]) and one that is suitable for image
classification (like [`BertForSequenceClassification`]).

As we mentioned before, we'll only write a loose wrapper of the model to keep it simple for this example. The only
thing we need to do before writing this class is a map between the block types and actual block classes. Then the
model is defined from the configuration by passing everything to the `ResNet` class:

```py
from transformers import PreTrainedModel
from timm.models.resnet import BasicBlock, Bottleneck, ResNet
from .configuration_resnet import ResnetConfig


BLOCK_MAPPING = {"basic": BasicBlock, "bottleneck": Bottleneck}


class ResnetModel(PreTrainedModel):
    config_class = ResnetConfig

    def __init__(self, config):
        super().__init__(config)
        block_layer = BLOCK_MAPPING[config.block_type]
        self.model = ResNet(
            block_layer,
            config.layers,
            num_classes=config.num_classes,
            in_chans=config.input_channels,
            cardinality=config.cardinality,
            base_width=config.base_width,
            stem_width=config.stem_width,
            stem_type=config.stem_type,
            avg_down=config.avg_down,
        )

    def forward(self, tensor):
        return self.model.forward_features(tensor)
```

For the model that will classify images, we just change the forward method:

```py
import torch


key: _node_content
string_value:
{"id_": "672d03d0-b1e5-4cad-9707-8223405a0550", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9eccb4f8-9be0-4b4f-b885-0425297592c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f2b284f6ad534dc2c03c12a7a88c3d4b7d0570b5d8b7a08892d964779e6d1930", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "89d70d17-62dc-43e0-9dad-53859d15bfde", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fbb20755430f805faa2eaca3ec923ed4fd1d0f4084de0487797f1f9b5ddbaa8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4f5fdbc-b6fe-48d9-ace6-6d452ed58847", "node_type": "1", "metadata": {}, "hash": "702567af40b8191685f370181cc253554298070bba59d092fc7a62c8ebfcae46", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4746, "end_char_idx": 8291, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Writing a custom model

Now that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that
extracts the hidden features from a batch of images (like [`BertModel`]) and one that is suitable for image
classification (like [`BertForSequenceClassification`]).

As we mentioned before, we'll only write a loose wrapper of the model to keep it simple for this example. The only
thing we need to do before writing this class is a map between the block types and actual block classes. Then the
model is defined from the configuration by passing everything to the `ResNet` class:

```py
from transformers import PreTrainedModel
from timm.models.resnet import BasicBlock, Bottleneck, ResNet
from .configuration_resnet import ResnetConfig


BLOCK_MAPPING = {"basic": BasicBlock, "bottleneck": Bottleneck}


class ResnetModel(PreTrainedModel):
    config_class = ResnetConfig

    def __init__(self, config):
        super().__init__(config)
        block_layer = BLOCK_MAPPING[config.block_type]
        self.model = ResNet(
            block_layer,
            config.layers,
            num_classes=config.num_classes,
            in_chans=config.input_channels,
            cardinality=config.cardinality,
            base_width=config.base_width,
            stem_width=config.stem_width,
            stem_type=config.stem_type,
            avg_down=config.avg_down,
        )

    def forward(self, tensor):
        return self.model.forward_features(tensor)
```

For the model that will classify images, we just change the forward method:

```py
import torch


class ResnetModelForImageClassification(PreTrainedModel):
    config_class = ResnetConfig

    def __init__(self, config):
        super().__init__(config)
        block_layer = BLOCK_MAPPING[config.block_type]
        self.model = ResNet(
            block_layer,
            config.layers,
            num_classes=config.num_classes,
            in_chans=config.input_channels,
            cardinality=config.cardinality,
            base_width=config.base_width,
            stem_width=config.stem_width,
            stem_type=config.stem_type,
            avg_down=config.avg_down,
        )

    def forward(self, tensor, labels=None):
        logits = self.model(tensor)
        if labels is not None:
            loss = torch.nn.cross_entropy(logits, labels)
            return {"loss": loss, "logits": logits}
        return {"logits": logits}
```

In both cases, notice how we inherit from `PreTrainedModel` and call the superclass initialization with the `config`
(a bit like when you write a regular `torch.nn.Module`). The line that sets the `config_class` is not mandatory, unless
you want to register your model with the auto classes (see last section).

<Tip>

If your model is very similar to a model inside the library, you can re-use the same configuration as this model.

</Tip>

You can have your model return anything you want, but returning a dictionary like we did for
`ResnetModelForImageClassification`, with the loss included when labels are passed, will make your model directly
usable inside the [`Trainer`] class. Using another output format is fine as long as you are planning on using your own
training loop or another library for training.

Now that we have our model class, let's create one:

```py
resnet50d = ResnetModelForImageClassification(resnet50d_config)
```

Again, you can use any of the methods of [`PreTrainedModel`], like [`~PreTrainedModel.save_pretrained`] or
[`~PreTrainedModel.push_to_hub`].


key: _node_content
string_value:
{"id_": "e4f5fdbc-b6fe-48d9-ace6-6d452ed58847", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9eccb4f8-9be0-4b4f-b885-0425297592c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f2b284f6ad534dc2c03c12a7a88c3d4b7d0570b5d8b7a08892d964779e6d1930", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "672d03d0-b1e5-4cad-9707-8223405a0550", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8cc1266513a2ece7353833c45df8c6bad2feb701a81c798c00ed2f19ff6c3fd6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d6ee8e84-b6c0-46a4-8ba7-ee661245fea7", "node_type": "1", "metadata": {}, "hash": "117424f2eb376189b1643b08affd0393634810d9fbe0256ba1afcf05b54ba6d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7381, "end_char_idx": 10738, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The line that sets the `config_class` is not mandatory, unless
you want to register your model with the auto classes (see last section).

<Tip>

If your model is very similar to a model inside the library, you can re-use the same configuration as this model.

</Tip>

You can have your model return anything you want, but returning a dictionary like we did for
`ResnetModelForImageClassification`, with the loss included when labels are passed, will make your model directly
usable inside the [`Trainer`] class. Using another output format is fine as long as you are planning on using your own
training loop or another library for training.

Now that we have our model class, let's create one:

```py
resnet50d = ResnetModelForImageClassification(resnet50d_config)
```

Again, you can use any of the methods of [`PreTrainedModel`], like [`~PreTrainedModel.save_pretrained`] or
[`~PreTrainedModel.push_to_hub`]. We will use the second in the next section, and see how to push the model weights
with the code of our model. But first, let's load some pretrained weights inside our model.

In your own use case, you will probably be training your custom model on your own data. To go fast for this tutorial,
we will use the pretrained version of the resnet50d. Since our model is just a wrapper around it, it's going to be
easy to transfer those weights:

```py
import timm

pretrained_model = timm.create_model("resnet50d", pretrained=True)
resnet50d.model.load_state_dict(pretrained_model.state_dict())
```

Now let's see how to make sure that when we do [`~PreTrainedModel.save_pretrained`] or [`~PreTrainedModel.push_to_hub`], the
code of the model is saved.

## Registering a model with custom code to the auto classes

If you are writing a library that extends 🤗 Transformers, you may want to extend the auto classes to include your own
model. This is different from pushing the code to the Hub in the sense that users will need to import your library to
get the custom models (contrarily to automatically downloading the model code from the Hub).

As long as your config has a `model_type` attribute that is different from existing model types, and that your model
classes have the right `config_class` attributes, you can just add them to the auto classes like this:

```py
from transformers import AutoConfig, AutoModel, AutoModelForImageClassification

AutoConfig.register("resnet", ResnetConfig)
AutoModel.register(ResnetConfig, ResnetModel)
AutoModelForImageClassification.register(ResnetConfig, ResnetModelForImageClassification)
```

Note that the first argument used when registering your custom config to [`AutoConfig`] needs to match the `model_type`
of your custom config, and the first argument used when registering your custom models to any auto model class needs
to match the `config_class` of those models.

## Sending the code to the Hub

<Tip warning={true}>

This API is experimental and may have some slight breaking changes in the next releases.

</Tip>

First, make sure your model is fully defined in a `.py` file. It can rely on relative imports to some other files as
long as all the files are in the same directory (we don't support submodules for this feature yet). For our example,
we'll define a `modeling_resnet.py` file and a `configuration_resnet.py` file in a folder of the current working
directory named `resnet_model`.


key: _node_content
string_value:
{"id_": "d6ee8e84-b6c0-46a4-8ba7-ee661245fea7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9eccb4f8-9be0-4b4f-b885-0425297592c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f2b284f6ad534dc2c03c12a7a88c3d4b7d0570b5d8b7a08892d964779e6d1930", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4f5fdbc-b6fe-48d9-ace6-6d452ed58847", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d16ac9712add1f00b601a4cecfd5c69a8052d6b211d37d3ce55f6d8801138bb1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8dceba1d-906c-44d5-ae7e-4dd80ae67938", "node_type": "1", "metadata": {}, "hash": "ed2c028b3ce8b65cf66d88bd536afe7300e259122596712e4ab91861ccecb80b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9041, "end_char_idx": 12465, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Registering a model with custom code to the auto classes

If you are writing a library that extends 🤗 Transformers, you may want to extend the auto classes to include your own
model. This is different from pushing the code to the Hub in the sense that users will need to import your library to
get the custom models (contrarily to automatically downloading the model code from the Hub).

As long as your config has a `model_type` attribute that is different from existing model types, and that your model
classes have the right `config_class` attributes, you can just add them to the auto classes like this:

```py
from transformers import AutoConfig, AutoModel, AutoModelForImageClassification

AutoConfig.register("resnet", ResnetConfig)
AutoModel.register(ResnetConfig, ResnetModel)
AutoModelForImageClassification.register(ResnetConfig, ResnetModelForImageClassification)
```

Note that the first argument used when registering your custom config to [`AutoConfig`] needs to match the `model_type`
of your custom config, and the first argument used when registering your custom models to any auto model class needs
to match the `config_class` of those models.

## Sending the code to the Hub

<Tip warning={true}>

This API is experimental and may have some slight breaking changes in the next releases.

</Tip>

First, make sure your model is fully defined in a `.py` file. It can rely on relative imports to some other files as
long as all the files are in the same directory (we don't support submodules for this feature yet). For our example,
we'll define a `modeling_resnet.py` file and a `configuration_resnet.py` file in a folder of the current working
directory named `resnet_model`. The configuration file contains the code for `ResnetConfig` and the modeling file
contains the code of `ResnetModel` and `ResnetModelForImageClassification`.

```
.
└── resnet_model
    ├── __init__.py
    ├── configuration_resnet.py
    └── modeling_resnet.py
```

The `__init__.py` can be empty, it's just there so that Python detects `resnet_model` can be use as a module.

<Tip warning={true}>

If copying a modeling files from the library, you will need to replace all the relative imports at the top of the file
to import from the `transformers` package.

</Tip>

Note that you can re-use (or subclass) an existing configuration/model.

To share your model with the community, follow those steps: first import the ResNet model and config from the newly
created files:

```py
from resnet_model.configuration_resnet import ResnetConfig
from resnet_model.modeling_resnet import ResnetModel, ResnetModelForImageClassification
```

Then you have to tell the library you want to copy the code files of those objects when using the `save_pretrained`
method and properly register them with a given Auto class (especially for models), just run:

```py
ResnetConfig.register_for_auto_class()
ResnetModel.register_for_auto_class("AutoModel")
ResnetModelForImageClassification.register_for_auto_class("AutoModelForImageClassification")
```

Note that there is no need to specify an auto class for the configuration (there is only one auto class for them,
[`AutoConfig`]) but it's different for models. Your custom model could be suitable for many different tasks, so you
have to specify which one of the auto classes is the correct one for your model.

<Tip>

Use `register_for_auto_class()` if you want the code files to be copied.


key: _node_content
string_value:
{"id_": "8dceba1d-906c-44d5-ae7e-4dd80ae67938", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9eccb4f8-9be0-4b4f-b885-0425297592c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f2b284f6ad534dc2c03c12a7a88c3d4b7d0570b5d8b7a08892d964779e6d1930", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d6ee8e84-b6c0-46a4-8ba7-ee661245fea7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "53a2b62d1efe6e454953534a527a45dc5db0e06e144503ecc8b342d2ad2c92b0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "114cb866-f357-40ae-a38e-8c6875a48beb", "node_type": "1", "metadata": {}, "hash": "0762a7b506e827b292dbf7086cd1992fd1d3acba3c252f22cbcd0b429e11cb15", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10739, "end_char_idx": 13922, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The configuration file contains the code for `ResnetConfig` and the modeling file
contains the code of `ResnetModel` and `ResnetModelForImageClassification`.

```
.
└── resnet_model
    ├── __init__.py
    ├── configuration_resnet.py
    └── modeling_resnet.py
```

The `__init__.py` can be empty, it's just there so that Python detects `resnet_model` can be use as a module.

<Tip warning={true}>

If copying a modeling files from the library, you will need to replace all the relative imports at the top of the file
to import from the `transformers` package.

</Tip>

Note that you can re-use (or subclass) an existing configuration/model.

To share your model with the community, follow those steps: first import the ResNet model and config from the newly
created files:

```py
from resnet_model.configuration_resnet import ResnetConfig
from resnet_model.modeling_resnet import ResnetModel, ResnetModelForImageClassification
```

Then you have to tell the library you want to copy the code files of those objects when using the `save_pretrained`
method and properly register them with a given Auto class (especially for models), just run:

```py
ResnetConfig.register_for_auto_class()
ResnetModel.register_for_auto_class("AutoModel")
ResnetModelForImageClassification.register_for_auto_class("AutoModelForImageClassification")
```

Note that there is no need to specify an auto class for the configuration (there is only one auto class for them,
[`AutoConfig`]) but it's different for models. Your custom model could be suitable for many different tasks, so you
have to specify which one of the auto classes is the correct one for your model.

<Tip>

Use `register_for_auto_class()` if you want the code files to be copied. If you instead prefer to use code on the Hub from another repo, 
you don't need to call it. In cases where there's more than one auto class, you can modify the `config.json` directly using the 
following structure:

```json
"auto_map": {     
	"AutoConfig": "<your-repo-name>--<config-name>",     
	"AutoModel": "<your-repo-name>--<config-name>",
	"AutoModelFor<Task>": "<your-repo-name>--<config-name>",    
},
```

</Tip>

Next, let's create the config and models as we did before:

```py
resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d = ResnetModelForImageClassification(resnet50d_config)

pretrained_model = timm.create_model("resnet50d", pretrained=True)
resnet50d.model.load_state_dict(pretrained_model.state_dict())
```

Now to send the model to the Hub, make sure you are logged in. Either run in your terminal:

```bash
huggingface-cli login
```

or from a notebook:

```py
from huggingface_hub import notebook_login

notebook_login()
```

You can then push to your own namespace (or an organization you are a member of) like this:

```py
resnet50d.push_to_hub("custom-resnet50d")
```

On top of the modeling weights and the configuration in json format, this also copied the modeling and
configuration `.py` files in the folder `custom-resnet50d` and uploaded the result to the Hub. You can check the result
in this [model repo](https://huggingface.co/sgugger/custom-resnet50d).


key: _node_content
string_value:
{"id_": "114cb866-f357-40ae-a38e-8c6875a48beb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9eccb4f8-9be0-4b4f-b885-0425297592c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f2b284f6ad534dc2c03c12a7a88c3d4b7d0570b5d8b7a08892d964779e6d1930", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8dceba1d-906c-44d5-ae7e-4dd80ae67938", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/custom_models/", "title": "Building custom models", "tokens": 3680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e422b96f6ee99ab96effc34ff0bf7b3925cf18837d786326171d98613c866217", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12466, "end_char_idx": 15285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you instead prefer to use code on the Hub from another repo, 
you don't need to call it. In cases where there's more than one auto class, you can modify the `config.json` directly using the 
following structure:

```json
"auto_map": {     
	"AutoConfig": "<your-repo-name>--<config-name>",     
	"AutoModel": "<your-repo-name>--<config-name>",
	"AutoModelFor<Task>": "<your-repo-name>--<config-name>",    
},
```

</Tip>

Next, let's create the config and models as we did before:

```py
resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d = ResnetModelForImageClassification(resnet50d_config)

pretrained_model = timm.create_model("resnet50d", pretrained=True)
resnet50d.model.load_state_dict(pretrained_model.state_dict())
```

Now to send the model to the Hub, make sure you are logged in. Either run in your terminal:

```bash
huggingface-cli login
```

or from a notebook:

```py
from huggingface_hub import notebook_login

notebook_login()
```

You can then push to your own namespace (or an organization you are a member of) like this:

```py
resnet50d.push_to_hub("custom-resnet50d")
```

On top of the modeling weights and the configuration in json format, this also copied the modeling and
configuration `.py` files in the folder `custom-resnet50d` and uploaded the result to the Hub. You can check the result
in this [model repo](https://huggingface.co/sgugger/custom-resnet50d).

See the [sharing tutorial](model_sharing) for more information on the push to Hub method.

## Using a model with custom code

You can use any configuration, model or tokenizer with custom code files in its repository with the auto-classes and
the `from_pretrained` method. All files and code uploaded to the Hub are scanned for malware (refer to the [Hub security](https://huggingface.co/docs/hub/security#malware-scanning) documentation for more information), but you should still 
review the model code and author to avoid executing malicious code on your machine. Set `trust_remote_code=True` to use
a model with custom code:

```py
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained("sgugger/custom-resnet50d", trust_remote_code=True)
```

It is also strongly encouraged to pass a commit hash as a `revision` to make sure the author of the models did not
update the code with some malicious new lines (unless you fully trust the authors of the models).

```py
commit_hash = "ed94a7c6247d8aedce4647f00f20de6875b5b292"
model = AutoModelForImageClassification.from_pretrained(
    "sgugger/custom-resnet50d", trust_remote_code=True, revision=commit_hash
)
```

Note that when browsing the commit history of the model repo on the Hub, there is a button to easily copy the commit
hash of any commit.


key: _node_content
string_value:
{"id_": "4e8ae44d-60b7-46cf-8e90-28e01264f232", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "41c81ccf-5080-4bd5-9d16-67048afadfcc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d6672a6084ede4c23086390dcb2bb897129abd7b007bcab9669d57e91b7aaeaa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5745c070-226e-44fa-9c79-8610813b95e2", "node_type": "1", "metadata": {}, "hash": "eee08c127d0da344197c7f0d0d399a0bf975872c44e75bc1bb990ea8f7485437", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2752, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Load pretrained instances with an AutoClass

With so many different Transformer architectures, it can be challenging to create one for your checkpoint. As a part of 🤗 Transformers core philosophy to make the library easy, simple and flexible to use, an `AutoClass` automatically infers and loads the correct architecture from a given checkpoint. The `from_pretrained()` method lets you quickly load a pretrained model for any architecture so you don't have to devote time and resources to train a model from scratch. Producing this type of checkpoint-agnostic code means if your code works for one checkpoint, it will work with another checkpoint - as long as it was trained for a similar task - even if the architecture is different.

<Tip>

Remember, architecture refers to the skeleton of the model and checkpoints are the weights for a given architecture. For example, [BERT](https://huggingface.co/google-bert/bert-base-uncased) is an architecture, while `google-bert/bert-base-uncased` is a checkpoint. Model is a general term that can mean either architecture or checkpoint.

</Tip>

In this tutorial, learn to:

* Load a pretrained tokenizer.
* Load a pretrained image processor
* Load a pretrained feature extractor.
* Load a pretrained processor.
* Load a pretrained model.
* Load a model as a backbone.

## AutoTokenizer

Nearly every NLP task begins with a tokenizer. A tokenizer converts your input into a format that can be processed by the model.

Load a tokenizer with [`AutoTokenizer.from_pretrained`]:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
```

Then tokenize your input as shown below:

```py
>>> sequence = "In a hole in the ground there lived a hobbit."
>>> print(tokenizer(sequence))
{'input_ids': [101, 1999, 1037, 4920, 1999, 1996, 2598, 2045, 2973, 1037, 7570, 10322, 4183, 1012, 102], 
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

## AutoImageProcessor

For vision tasks, an image processor processes the image into the correct input format.

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
```

## AutoBackbone

<div style="text-align: center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stages.png">
    <figcaption class="mt-2 text-center text-sm text-gray-500">A Swin backbone with multiple stages for outputting a feature map.</figcaption>
</div>

The [`AutoBackbone`] lets you use pretrained models as backbones to get feature maps from different stages of the backbone.


key: _node_content
string_value:
{"id_": "5745c070-226e-44fa-9c79-8610813b95e2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "41c81ccf-5080-4bd5-9d16-67048afadfcc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d6672a6084ede4c23086390dcb2bb897129abd7b007bcab9669d57e91b7aaeaa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4e8ae44d-60b7-46cf-8e90-28e01264f232", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1b46702c48530ec59c4cdd2a15b35967f88b5a1651e9e96d7ddfb2288d09eac2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3a7600b-1b07-46e9-b37b-3f905d8464dc", "node_type": "1", "metadata": {}, "hash": "28f80ca76a2be400d18f73413702348ada1cccdb895d9037527582d29e9b9a5e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1768, "end_char_idx": 4537, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> print(tokenizer(sequence))
{'input_ids': [101, 1999, 1037, 4920, 1999, 1996, 2598, 2045, 2973, 1037, 7570, 10322, 4183, 1012, 102], 
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

## AutoImageProcessor

For vision tasks, an image processor processes the image into the correct input format.

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
```

## AutoBackbone

<div style="text-align: center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stages.png">
    <figcaption class="mt-2 text-center text-sm text-gray-500">A Swin backbone with multiple stages for outputting a feature map.</figcaption>
</div>

The [`AutoBackbone`] lets you use pretrained models as backbones to get feature maps from different stages of the backbone. You should specify one of the following parameters in [`~PretrainedConfig.from_pretrained`]:

* `out_indices` is the index of the layer you'd like to get the feature map from
* `out_features` is the name of the layer you'd like to get the feature map from

These parameters can be used interchangeably, but if you use both, make sure they're aligned with each other! If you don't pass any of these parameters, the backbone returns the feature map from the last layer.

<div style="text-align: center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stage%201.png">
    <figcaption class="mt-2 text-center text-sm text-gray-500">A feature map from the first stage of the backbone. The patch partition refers to the model stem.</figcaption>
</div>

For example, in the above diagram, to return the feature map from the first stage of the Swin backbone, you can set `out_indices=(1,)`:

```py
>>> from transformers import AutoImageProcessor, AutoBackbone
>>> import torch
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> processor = AutoImageProcessor.from_pretrained("microsoft/swin-tiny-patch4-window7-224")
>>> model = AutoBackbone.from_pretrained("microsoft/swin-tiny-patch4-window7-224", out_indices=(1,))

>>> inputs = processor(image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> feature_maps = outputs.feature_maps
```

Now you can access the `feature_maps` object from the first stage of the backbone:

```py
>>> list(feature_maps[0].shape)
[1, 96, 56, 56]
```

## AutoFeatureExtractor

For audio tasks, a feature extractor processes the audio signal the correct input format.


key: _node_content
string_value:
{"id_": "d3a7600b-1b07-46e9-b37b-3f905d8464dc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "41c81ccf-5080-4bd5-9d16-67048afadfcc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d6672a6084ede4c23086390dcb2bb897129abd7b007bcab9669d57e91b7aaeaa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5745c070-226e-44fa-9c79-8610813b95e2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1367fae8e6df968585b076a6e5f66faeb517a9887e7668c6da91698a7bc1c2d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bc3f69a4-4190-4c6f-8f96-99ca5290a260", "node_type": "1", "metadata": {}, "hash": "381199319bccdd20b339d58e6c1dce48138341d80a3e5d9655cf689f789194ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3120, "end_char_idx": 6276, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you don't pass any of these parameters, the backbone returns the feature map from the last layer.

<div style="text-align: center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Swin%20Stage%201.png">
    <figcaption class="mt-2 text-center text-sm text-gray-500">A feature map from the first stage of the backbone. The patch partition refers to the model stem.</figcaption>
</div>

For example, in the above diagram, to return the feature map from the first stage of the Swin backbone, you can set `out_indices=(1,)`:

```py
>>> from transformers import AutoImageProcessor, AutoBackbone
>>> import torch
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> processor = AutoImageProcessor.from_pretrained("microsoft/swin-tiny-patch4-window7-224")
>>> model = AutoBackbone.from_pretrained("microsoft/swin-tiny-patch4-window7-224", out_indices=(1,))

>>> inputs = processor(image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> feature_maps = outputs.feature_maps
```

Now you can access the `feature_maps` object from the first stage of the backbone:

```py
>>> list(feature_maps[0].shape)
[1, 96, 56, 56]
```

## AutoFeatureExtractor

For audio tasks, a feature extractor processes the audio signal the correct input format.

Load a feature extractor with [`AutoFeatureExtractor.from_pretrained`]:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained(
...     "ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition"
... )
```

## AutoProcessor

Multimodal tasks require a processor that combines two types of preprocessing tools. For example, the [LayoutLMV2](model_doc/layoutlmv2) model requires an image processor to handle images and a tokenizer to handle text; a processor combines both of them.

Load a processor with [`AutoProcessor.from_pretrained`]:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")
```

## AutoModel

<frameworkcontent>
<pt>
The `AutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks). For example, load a model for sequence classification with [`AutoModelForSequenceClassification.from_pretrained`]:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse the same checkpoint to load an architecture for a different task:

```py
>>> from transformers import AutoModelForTokenClassification

>>> model = AutoModelForTokenClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

<Tip warning={true}>

For PyTorch models, the `from_pretrained()` method uses `torch.load()` which internally uses `pickle` and is known to be insecure. In general, never load a model that could have come from an untrusted source, or that could have been tampered with.


key: _node_content
string_value:
{"id_": "bc3f69a4-4190-4c6f-8f96-99ca5290a260", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "41c81ccf-5080-4bd5-9d16-67048afadfcc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d6672a6084ede4c23086390dcb2bb897129abd7b007bcab9669d57e91b7aaeaa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d3a7600b-1b07-46e9-b37b-3f905d8464dc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c57822889e565d9d3639404412ace41baab355c17390eeecffca741e1d05a2a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b05bbb29-b18e-4158-b767-d23483e38b7a", "node_type": "1", "metadata": {}, "hash": "54ab0752e326d3f02721787158ef44640b3c6cf1eb3e6d64ded4177f27aeb082", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4539, "end_char_idx": 7409, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Load a feature extractor with [`AutoFeatureExtractor.from_pretrained`]:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained(
...     "ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition"
... )
```

## AutoProcessor

Multimodal tasks require a processor that combines two types of preprocessing tools. For example, the [LayoutLMV2](model_doc/layoutlmv2) model requires an image processor to handle images and a tokenizer to handle text; a processor combines both of them.

Load a processor with [`AutoProcessor.from_pretrained`]:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")
```

## AutoModel

<frameworkcontent>
<pt>
The `AutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks). For example, load a model for sequence classification with [`AutoModelForSequenceClassification.from_pretrained`]:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse the same checkpoint to load an architecture for a different task:

```py
>>> from transformers import AutoModelForTokenClassification

>>> model = AutoModelForTokenClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

<Tip warning={true}>

For PyTorch models, the `from_pretrained()` method uses `torch.load()` which internally uses `pickle` and is known to be insecure. In general, never load a model that could have come from an untrusted source, or that could have been tampered with. This security risk is partially mitigated for public models hosted on the Hugging Face Hub, which are [scanned for malware](https://huggingface.co/docs/hub/security-malware) at each commit. See the [Hub documentation](https://huggingface.co/docs/hub/security) for best practices like [signed commit verification](https://huggingface.co/docs/hub/security-gpg#signing-commits-with-gpg) with GPG.

TensorFlow and Flax checkpoints are not affected, and can be loaded within PyTorch architectures using the `from_tf` and `from_flax` kwargs for the `from_pretrained` method to circumvent this issue.

</Tip>

Generally, we recommend using the `AutoTokenizer` class and the `AutoModelFor` class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next [tutorial](preprocessing), learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.
</pt>
<tf>
Finally, the `TFAutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks).


key: _node_content
string_value:
{"id_": "b05bbb29-b18e-4158-b767-d23483e38b7a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "41c81ccf-5080-4bd5-9d16-67048afadfcc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d6672a6084ede4c23086390dcb2bb897129abd7b007bcab9669d57e91b7aaeaa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc3f69a4-4190-4c6f-8f96-99ca5290a260", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/autoclass_tutorial/", "title": "Load pretrained instances with an AutoClass", "tokens": 2150, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e12157b8e414582527d5b88598b781fa4133e4bab07d49fd9dd1e90496996e58", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6160, "end_char_idx": 8354, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In general, never load a model that could have come from an untrusted source, or that could have been tampered with. This security risk is partially mitigated for public models hosted on the Hugging Face Hub, which are [scanned for malware](https://huggingface.co/docs/hub/security-malware) at each commit. See the [Hub documentation](https://huggingface.co/docs/hub/security) for best practices like [signed commit verification](https://huggingface.co/docs/hub/security-gpg#signing-commits-with-gpg) with GPG.

TensorFlow and Flax checkpoints are not affected, and can be loaded within PyTorch architectures using the `from_tf` and `from_flax` kwargs for the `from_pretrained` method to circumvent this issue.

</Tip>

Generally, we recommend using the `AutoTokenizer` class and the `AutoModelFor` class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next [tutorial](preprocessing), learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.
</pt>
<tf>
Finally, the `TFAutoModelFor` classes let you load a pretrained model for a given task (see [here](model_doc/auto) for a complete list of available tasks). For example, load a model for sequence classification with [`TFAutoModelForSequenceClassification.from_pretrained`]:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse the same checkpoint to load an architecture for a different task:

```py
>>> from transformers import TFAutoModelForTokenClassification

>>> model = TFAutoModelForTokenClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Generally, we recommend using the `AutoTokenizer` class and the `TFAutoModelFor` class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next [tutorial](preprocessing), learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "0387e517-943c-4777-9a23-4099d3ba863f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "281e674f-faba-483d-9856-1aea7421fd5c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0d4b18e02d9032d1fcc56068adec3eac44e17d6bb448695fa21995882f0f7be5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "643e26a9-5b2e-464d-99bb-fae628a57d79", "node_type": "1", "metadata": {}, "hash": "4e29a365cf288d80c7dab262266ef79d8dbebbd54f2eec6e1ee4c9927ab70a31", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3101, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CPU inference

With some optimizations, it is possible to efficiently run large model inference on a CPU. One of these optimization techniques involves compiling the PyTorch code into an intermediate format for high-performance environments like C++. The other technique fuses multiple operations into one kernel to reduce the overhead of running each operation separately.

You'll learn how to use [BetterTransformer](https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/) for faster inference, and how to convert your PyTorch code to [TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html). If you're using an Intel CPU, you can also use [graph optimizations](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features.html#graph-optimization) from [Intel Extension for PyTorch](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/index.html) to boost inference speed even more. Finally, learn how to use 🤗 Optimum to accelerate inference with ONNX Runtime or OpenVINO (if you're using an Intel CPU).

## BetterTransformer

BetterTransformer accelerates inference with its fastpath (native PyTorch specialized implementation of Transformer functions) execution. The two optimizations in the fastpath execution are:

1. fusion, which combines multiple sequential operations into a single "kernel" to reduce the number of computation steps
2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors

BetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention).

<Tip>

BetterTransformer is not supported for all models. Check this [list](https://huggingface.co/docs/optimum/bettertransformer/overview#supported-models) to see if a model supports BetterTransformer.

</Tip>

Before you start, make sure you have 🤗 Optimum [installed](https://huggingface.co/docs/optimum/installation).

Enable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigcode/starcoder")
model.to_bettertransformer()
```

## TorchScript

TorchScript is an intermediate PyTorch model representation that can be run in production environments where performance is important. You can train a model in PyTorch and then export it to TorchScript to free the model from Python performance constraints. PyTorch [traces](https://pytorch.org/docs/stable/generated/torch.jit.trace.html) a model to return a [`ScriptFunction`] that is optimized with just-in-time compilation (JIT). Compared to the default eager mode, JIT mode in PyTorch typically yields better performance for inference using optimization techniques like operator fusion.

For a gentle introduction to TorchScript, see the [Introduction to PyTorch TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) tutorial.


key: _node_content
string_value:
{"id_": "643e26a9-5b2e-464d-99bb-fae628a57d79", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "281e674f-faba-483d-9856-1aea7421fd5c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0d4b18e02d9032d1fcc56068adec3eac44e17d6bb448695fa21995882f0f7be5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0387e517-943c-4777-9a23-4099d3ba863f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d5646bfd3c8bae4899227eca4941e794c4df068a9005dbd54deed7d282f1975", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0735fcc2-1b5c-4778-9779-04e0c32e8c81", "node_type": "1", "metadata": {}, "hash": "17a3c268249194838a77a1e6aaf373a827a44c6d8a8bab35da0464c6bd50c7a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1265, "end_char_idx": 4457, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The two optimizations in the fastpath execution are:

1. fusion, which combines multiple sequential operations into a single "kernel" to reduce the number of computation steps
2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors

BetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention).

<Tip>

BetterTransformer is not supported for all models. Check this [list](https://huggingface.co/docs/optimum/bettertransformer/overview#supported-models) to see if a model supports BetterTransformer.

</Tip>

Before you start, make sure you have 🤗 Optimum [installed](https://huggingface.co/docs/optimum/installation).

Enable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigcode/starcoder")
model.to_bettertransformer()
```

## TorchScript

TorchScript is an intermediate PyTorch model representation that can be run in production environments where performance is important. You can train a model in PyTorch and then export it to TorchScript to free the model from Python performance constraints. PyTorch [traces](https://pytorch.org/docs/stable/generated/torch.jit.trace.html) a model to return a [`ScriptFunction`] that is optimized with just-in-time compilation (JIT). Compared to the default eager mode, JIT mode in PyTorch typically yields better performance for inference using optimization techniques like operator fusion.

For a gentle introduction to TorchScript, see the [Introduction to PyTorch TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) tutorial.

With the [`Trainer`] class, you can enable JIT mode for CPU inference by setting the `--jit_mode_eval` flag:

```bash
python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--jit_mode_eval
```

<Tip warning={true}>

For PyTorch >= 1.14.0, JIT-mode could benefit any model for prediction and evaluation since the dict input is supported in `jit.trace`.

For PyTorch < 1.14.0, JIT-mode could benefit a model if its forward parameter order matches the tuple input order in `jit.trace`, such as a question-answering model. If the forward parameter order does not match the tuple input order in `jit.trace`, like a text classification model, `jit.trace` will fail and we are capturing this with the exception here to make it fallback. Logging is used to notify users.

</Tip>

## IPEX graph optimization

Intel® Extension for PyTorch (IPEX) provides further optimizations in JIT mode for Intel CPUs, and we recommend combining it with TorchScript for even faster performance. The IPEX [graph optimization](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/graph_optimization.html) fuses operations like Multi-head attention, Concat Linear, Linear + Add, Linear + Gelu, Add + LayerNorm, and more.


key: _node_content
string_value:
{"id_": "0735fcc2-1b5c-4778-9779-04e0c32e8c81", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "281e674f-faba-483d-9856-1aea7421fd5c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0d4b18e02d9032d1fcc56068adec3eac44e17d6bb448695fa21995882f0f7be5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "643e26a9-5b2e-464d-99bb-fae628a57d79", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3e2c4315b15d222a69ad628e1b91e055dc324d39b6e1445f7dbad90f90c12fe7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5d41119-6cc3-4dee-9ccb-5f5ab6092a8f", "node_type": "1", "metadata": {}, "hash": "264dbe468f53d819287f59c7b8637853f250e1a7d7872a587eb70aa3d07a9945", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2929, "end_char_idx": 5674, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For a gentle introduction to TorchScript, see the [Introduction to PyTorch TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) tutorial.

With the [`Trainer`] class, you can enable JIT mode for CPU inference by setting the `--jit_mode_eval` flag:

```bash
python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--jit_mode_eval
```

<Tip warning={true}>

For PyTorch >= 1.14.0, JIT-mode could benefit any model for prediction and evaluation since the dict input is supported in `jit.trace`.

For PyTorch < 1.14.0, JIT-mode could benefit a model if its forward parameter order matches the tuple input order in `jit.trace`, such as a question-answering model. If the forward parameter order does not match the tuple input order in `jit.trace`, like a text classification model, `jit.trace` will fail and we are capturing this with the exception here to make it fallback. Logging is used to notify users.

</Tip>

## IPEX graph optimization

Intel® Extension for PyTorch (IPEX) provides further optimizations in JIT mode for Intel CPUs, and we recommend combining it with TorchScript for even faster performance. The IPEX [graph optimization](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/graph_optimization.html) fuses operations like Multi-head attention, Concat Linear, Linear + Add, Linear + Gelu, Add + LayerNorm, and more.

To take advantage of these graph optimizations, make sure you have IPEX [installed](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/installation.html):

```bash
pip install intel_extension_for_pytorch
```

Set the `--use_ipex` and `--jit_mode_eval` flags in the [`Trainer`] class to enable JIT mode with the graph optimizations:

```bash
python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--use_ipex \
--jit_mode_eval
```

## 🤗 Optimum

<Tip>

Learn more details about using ORT with 🤗 Optimum in the [Optimum Inference with ONNX Runtime](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models) guide. This section only provides a brief and simple example.

</Tip>

ONNX Runtime (ORT) is a model accelerator that runs inference on CPUs by default. ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers, without making too many changes to your code. You only need to replace the 🤗 Transformers `AutoClass` with its equivalent [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and load a checkpoint in the ONNX format.


key: _node_content
string_value:
{"id_": "e5d41119-6cc3-4dee-9ccb-5f5ab6092a8f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "281e674f-faba-483d-9856-1aea7421fd5c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0d4b18e02d9032d1fcc56068adec3eac44e17d6bb448695fa21995882f0f7be5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0735fcc2-1b5c-4778-9779-04e0c32e8c81", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_cpu/", "title": "CPU inference", "tokens": 1709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4d3b6b510476dd03de683bacd749c568d46d578ab4991cd6f31237bde58036ad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4206, "end_char_idx": 6711, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The IPEX [graph optimization](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/graph_optimization.html) fuses operations like Multi-head attention, Concat Linear, Linear + Add, Linear + Gelu, Add + LayerNorm, and more.

To take advantage of these graph optimizations, make sure you have IPEX [installed](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/installation.html):

```bash
pip install intel_extension_for_pytorch
```

Set the `--use_ipex` and `--jit_mode_eval` flags in the [`Trainer`] class to enable JIT mode with the graph optimizations:

```bash
python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--use_ipex \
--jit_mode_eval
```

## 🤗 Optimum

<Tip>

Learn more details about using ORT with 🤗 Optimum in the [Optimum Inference with ONNX Runtime](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models) guide. This section only provides a brief and simple example.

</Tip>

ONNX Runtime (ORT) is a model accelerator that runs inference on CPUs by default. ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers, without making too many changes to your code. You only need to replace the 🤗 Transformers `AutoClass` with its equivalent [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and load a checkpoint in the ONNX format.

For example, if you're running inference on a question answering task, load the [optimum/roberta-base-squad2](https://huggingface.co/optimum/roberta-base-squad2) checkpoint which contains a `model.onnx` file:

```py
from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForQuestionAnswering

model = ORTModelForQuestionAnswering.from_pretrained("optimum/roberta-base-squad2")
tokenizer = AutoTokenizer.from_pretrained("deepset/roberta-base-squad2")

onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer)

question = "What's my name?"
context = "My name is Philipp and I live in Nuremberg."
pred = onnx_qa(question, context)
```

If you have an Intel CPU, take a look at 🤗 [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) which supports a variety of compression techniques (quantization, pruning, knowledge distillation) and tools for converting models to the [OpenVINO](https://huggingface.co/docs/optimum/intel/inference) format for higher performance inference.


key: _node_content
string_value:
{"id_": "ff5a9475-a3c7-441d-b4e9-3f3658a60c69", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/bertology/", "title": "BERTology", "tokens": 562, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "359ea2c6-c35a-4679-968f-3033c714a04b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/bertology/", "title": "BERTology", "tokens": 562, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3ef2bc3fadb40a5697e13a2d394482e70badd89a9421438a4e88f55b4e855aa6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1576, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BERTology

There is a growing field of study concerned with investigating the inner working of large-scale transformers like BERT
(that some call "BERTology"). Some good examples of this field are:


- BERT Rediscovers the Classical NLP Pipeline by Ian Tenney, Dipanjan Das, Ellie Pavlick:
  https://arxiv.org/abs/1905.05950
- Are Sixteen Heads Really Better than One? by Paul Michel, Omer Levy, Graham Neubig: https://arxiv.org/abs/1905.10650
- What Does BERT Look At? An Analysis of BERT's Attention by Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D.
  Manning: https://arxiv.org/abs/1906.04341
- CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure: https://arxiv.org/abs/2210.04633

In order to help this new field develop, we have included a few additional features in the BERT/GPT/GPT-2 models to
help people access the inner representations, mainly adapted from the great work of Paul Michel
(https://arxiv.org/abs/1905.10650):


- accessing all the hidden-states of BERT/GPT/GPT-2,
- accessing all the attention weights for each head of BERT/GPT/GPT-2,
- retrieving heads output values and gradients to be able to compute head importance score and prune head as explained
  in https://arxiv.org/abs/1905.10650.

To help you understand and use these features, we have added a specific example script: [bertology.py](https://github.com/huggingface/transformers/tree/main/examples/research_projects/bertology/run_bertology.py) while extract information and prune a model pre-trained on
GLUE.


key: _node_content
string_value:
{"id_": "4cf8822a-ba1d-464b-9c94-d4cfb048138e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4cc6bf8-a490-49f5-b337-84b7a8e4b834", "node_type": "1", "metadata": {}, "hash": "d2699a1a364d3677a2084a1a0bde7a9e246ccda461215d92ed5ac6ef92bb8a82", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2378, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Testing


Let's take a look at how 🤗 Transformers models are tested and how you can write new tests and improve the existing ones.

There are 2 test suites in the repository:

1. `tests` -- tests for the general API
2. `examples` -- tests primarily for various applications that aren't part of the API

## How transformers are tested

1. Once a PR is submitted it gets tested with 9 CircleCi jobs. Every new commit to that PR gets retested. These jobs
   are defined in this [config file](https://github.com/huggingface/transformers/tree/main/.circleci/config.yml), so that if needed you can reproduce the same
   environment on your machine.

   These CI jobs don't run `@slow` tests.

2. There are 3 jobs run by [github actions](https://github.com/huggingface/transformers/actions):

   - [torch hub integration](https://github.com/huggingface/transformers/tree/main/.github/workflows/github-torch-hub.yml): checks whether torch hub
     integration works.

   - [self-hosted (push)](https://github.com/huggingface/transformers/tree/main/.github/workflows/self-push.yml): runs fast tests on GPU only on commits on
     `main`. It only runs if a commit on `main` has updated the code in one of the following folders: `src`,
     `tests`, `.github` (to prevent running on added model cards, notebooks, etc.)

   - [self-hosted runner](https://github.com/huggingface/transformers/tree/main/.github/workflows/self-scheduled.yml): runs normal and slow tests on GPU in
     `tests` and `examples`:

```bash
RUN_SLOW=1 pytest tests/
RUN_SLOW=1 pytest examples/
```

   The results can be observed [here](https://github.com/huggingface/transformers/actions).



## Running tests



### Choosing which tests to run

This document goes into many details of how tests can be run. If after reading everything, you need even more details
you will find them [here](https://docs.pytest.org/en/latest/usage.html).

Here are some most useful ways of running tests.

Run all:

```console
pytest
```

or:

```bash
make test
```

Note that the latter is defined as:

```bash
python -m pytest -n auto --dist=loadfile -s -v ./tests/
```

which tells pytest to:

- run as many test processes as they are CPU cores (which could be too many if you don't have a ton of RAM!)
- ensure that all tests from the same file will be run by the same test process
- do not capture output
- run in verbose mode


key: _node_content
string_value:
{"id_": "e4cc6bf8-a490-49f5-b337-84b7a8e4b834", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4cf8822a-ba1d-464b-9c94-d4cfb048138e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "efa88c17849af560d963153cb3c26906f19b84446f54a735c97153c4e3f8a54d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "819b3660-3e10-4b5e-88a0-ed04fc93124a", "node_type": "1", "metadata": {}, "hash": "19444e82403b435ed353821215c57c13d8d52893289bf36176c0c2cddb1a6f45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1658, "end_char_idx": 4733, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Running tests



### Choosing which tests to run

This document goes into many details of how tests can be run. If after reading everything, you need even more details
you will find them [here](https://docs.pytest.org/en/latest/usage.html).

Here are some most useful ways of running tests.

Run all:

```console
pytest
```

or:

```bash
make test
```

Note that the latter is defined as:

```bash
python -m pytest -n auto --dist=loadfile -s -v ./tests/
```

which tells pytest to:

- run as many test processes as they are CPU cores (which could be too many if you don't have a ton of RAM!)
- ensure that all tests from the same file will be run by the same test process
- do not capture output
- run in verbose mode



### Getting the list of all tests

All tests of the test suite:

```bash
pytest --collect-only -q
```

All tests of a given test file:

```bash
pytest tests/test_optimization.py --collect-only -q
```

### Run a specific test module

To run an individual test module:

```bash
pytest tests/utils/test_logging.py
```

### Run specific tests

Since unittest is used inside most of the tests, to run specific subtests you need to know the name of the unittest
class containing those tests. For example, it could be:

```bash
pytest tests/test_optimization.py::OptimizationTest::test_adam_w
```

Here:

- `tests/test_optimization.py` - the file with tests
- `OptimizationTest` - the name of the class
- `test_adam_w` - the name of the specific test function

If the file contains multiple classes, you can choose to run only tests of a given class. For example:

```bash
pytest tests/test_optimization.py::OptimizationTest
```

will run all the tests inside that class.

As mentioned earlier you can see what tests are contained inside the `OptimizationTest` class by running:

```bash
pytest tests/test_optimization.py::OptimizationTest --collect-only -q
```

You can run tests by keyword expressions.

To run only tests whose name contains `adam`:

```bash
pytest -k adam tests/test_optimization.py
```

Logical `and` and `or` can be used to indicate whether all keywords should match or either. `not` can be used to
negate.

To run all tests except those whose name contains `adam`:

```bash
pytest -k "not adam" tests/test_optimization.py
```

And you can combine the two patterns in one:

```bash
pytest -k "ada and not adam" tests/test_optimization.py
```

For example to run both `test_adafactor` and `test_adam_w` you can use:

```bash
pytest -k "test_adafactor or test_adam_w" tests/test_optimization.py
```

Note that we use `or` here, since we want either of the keywords to match to include both.

If you want to include only tests that include both patterns, `and` is to be used:

```bash
pytest -k "test and ada" tests/test_optimization.py
```

### Run `accelerate` tests

Sometimes you need to run `accelerate` tests on your models. For that you can just add `-m accelerate_tests` to your command, if let's say you want to run these tests on `OPT` run:

```bash
RUN_SLOW=1 pytest -m accelerate_tests tests/models/opt/test_modeling_opt.py
```


key: _node_content
string_value:
{"id_": "819b3660-3e10-4b5e-88a0-ed04fc93124a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4cc6bf8-a490-49f5-b337-84b7a8e4b834", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "1fcd0cd680905cfa2bd626c461eea29250fd6d454a263cd4cbe3f481583403f6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aafe7d8a-623c-4cb9-a040-71a4bb33bacc", "node_type": "1", "metadata": {}, "hash": "f974b6090a54b5dbd7073ab95b8f0d8e220bfcb8ace9a39e6607283df322a180", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4736, "end_char_idx": 7839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Run documentation tests

In order to test whether the documentation examples are correct, you should check that the `doctests` are passing.
As an example, let's use [`WhisperModel.forward`'s docstring](https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/modeling_whisper.py#L1017-L1035):

```python
r"""
Returns:

Example:
    ```python
    >>> import torch
    >>> from transformers import WhisperModel, WhisperFeatureExtractor
    >>> from datasets import load_dataset

    >>> model = WhisperModel.from_pretrained("openai/whisper-base")
    >>> feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-base")
    >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
    >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt")
    >>> input_features = inputs.input_features
    >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id
    >>> last_hidden_state = model(input_features, decoder_input_ids=decoder_input_ids).last_hidden_state
    >>> list(last_hidden_state.shape)
    [1, 2, 512]
    ```"""

```

Just run the following line to automatically test every docstring example in the desired file:
```bash
pytest --doctest-modules <path_to_file_or_dir>
```
If the file has a markdown extention, you should add the `--doctest-glob="*.md"` argument.

### Run only modified tests

You can run the tests related to the unstaged files or the current branch (according to Git) by using [pytest-picked](https://github.com/anapaulagomes/pytest-picked). This is a great way of quickly testing your changes didn't break
anything, since it won't run the tests related to files you didn't touch.

```bash
pip install pytest-picked
```

```bash
pytest --picked
```

All tests will be run from files and folders which are modified, but not yet committed.

### Automatically rerun failed tests on source modification

[pytest-xdist](https://github.com/pytest-dev/pytest-xdist) provides a very useful feature of detecting all failed
tests, and then waiting for you to modify files and continuously re-rerun those failing tests until they pass while you
fix them. So that you don't need to re start pytest after you made the fix. This is repeated until all tests pass after
which again a full run is performed.

```bash
pip install pytest-xdist
```

To enter the mode: `pytest -f` or `pytest --looponfail`

File changes are detected by looking at `looponfailroots` root directories and all of their contents (recursively).
If the default for this value does not work for you, you can change it in your project by setting a configuration
option in `setup.cfg`:

```ini
[tool:pytest]
looponfailroots = transformers tests
```

or `pytest.ini`/``tox.ini`` files:

```ini
[pytest]
looponfailroots = transformers tests
```

This would lead to only looking for file changes in the respective directories, specified relatively to the ini-file’s
directory.

[pytest-watch](https://github.com/joeyespo/pytest-watch) is an alternative implementation of this functionality.


key: _node_content
string_value:
{"id_": "aafe7d8a-623c-4cb9-a040-71a4bb33bacc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "819b3660-3e10-4b5e-88a0-ed04fc93124a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "fe54436ec5a360a41ba166c5c08171230de63e5d1257ed78ba3d7b8219d21ad2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "10a2f61b-b843-40a6-896b-4436b347964c", "node_type": "1", "metadata": {}, "hash": "917bb5093723d0630fe5e55dbd8341796a235f3296f05bbec98762e0850b3393", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7842, "end_char_idx": 9244, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Skip a test module

If you want to run all test modules, except a few you can exclude them by giving an explicit list of tests to run. For
example, to run all except `test_modeling_*.py` tests:

```bash
pytest *ls -1 tests/*py | grep -v test_modeling*
```

### Clearing state

CI builds and when isolation is important (against speed), cache should be cleared:

```bash
pytest --cache-clear tests
```

### Running tests in parallel

As mentioned earlier `make test` runs tests in parallel via `pytest-xdist` plugin (`-n X` argument, e.g. `-n 2`
to run 2 parallel jobs).

`pytest-xdist`'s `--dist=` option allows one to control how the tests are grouped. `--dist=loadfile` puts the
tests located in one file onto the same process.

Since the order of executed tests is different and unpredictable, if running the test suite with `pytest-xdist`
produces failures (meaning we have some undetected coupled tests), use [pytest-replay](https://github.com/ESSS/pytest-replay) to replay the tests in the same order, which should help with then somehow
reducing that failing sequence to a minimum.

### Test order and repetition

It's good to repeat the tests several times, in sequence, randomly, or in sets, to detect any potential
inter-dependency and state-related bugs (tear down). And the straightforward multiple repetition is just good to detect
some problems that get uncovered by randomness of DL.


key: _node_content
string_value:
{"id_": "10a2f61b-b843-40a6-896b-4436b347964c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aafe7d8a-623c-4cb9-a040-71a4bb33bacc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "a3d6480b96397944f029d0d24aa448257d01022b25e13107a160a27719867cea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b89107d2-4568-48fc-83db-1033f6272fda", "node_type": "1", "metadata": {}, "hash": "7a00815b5ed9a5e98bbbdd173f6c73689aad44eea906d406e7b59362c56f07de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7842, "end_char_idx": 11942, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Skip a test module

If you want to run all test modules, except a few you can exclude them by giving an explicit list of tests to run. For
example, to run all except `test_modeling_*.py` tests:

```bash
pytest *ls -1 tests/*py | grep -v test_modeling*
```

### Clearing state

CI builds and when isolation is important (against speed), cache should be cleared:

```bash
pytest --cache-clear tests
```

### Running tests in parallel

As mentioned earlier `make test` runs tests in parallel via `pytest-xdist` plugin (`-n X` argument, e.g. `-n 2`
to run 2 parallel jobs).

`pytest-xdist`'s `--dist=` option allows one to control how the tests are grouped. `--dist=loadfile` puts the
tests located in one file onto the same process.

Since the order of executed tests is different and unpredictable, if running the test suite with `pytest-xdist`
produces failures (meaning we have some undetected coupled tests), use [pytest-replay](https://github.com/ESSS/pytest-replay) to replay the tests in the same order, which should help with then somehow
reducing that failing sequence to a minimum.

### Test order and repetition

It's good to repeat the tests several times, in sequence, randomly, or in sets, to detect any potential
inter-dependency and state-related bugs (tear down). And the straightforward multiple repetition is just good to detect
some problems that get uncovered by randomness of DL.


#### Repeat tests

- [pytest-flakefinder](https://github.com/dropbox/pytest-flakefinder):

```bash
pip install pytest-flakefinder
```

And then run every test multiple times (50 by default):

```bash
pytest --flake-finder --flake-runs=5 tests/test_failing_test.py
```

<Tip>

This plugin doesn't work with `-n` flag from `pytest-xdist`.

</Tip>

<Tip>

There is another plugin `pytest-repeat`, but it doesn't work with `unittest`.

</Tip>

#### Run tests in a random order

```bash
pip install pytest-random-order
```

Important: the presence of `pytest-random-order` will automatically randomize tests, no configuration change or
command line options is required.

As explained earlier this allows detection of coupled tests - where one test's state affects the state of another. When
`pytest-random-order` is installed it will print the random seed it used for that session, e.g:

```bash
pytest tests
[...]
Using --random-order-bucket=module
Using --random-order-seed=573663
```

So that if the given particular sequence fails, you can reproduce it by adding that exact seed, e.g.:

```bash
pytest --random-order-seed=573663
[...]
Using --random-order-bucket=module
Using --random-order-seed=573663
```

It will only reproduce the exact order if you use the exact same list of tests (or no list at all). Once you start to
manually narrowing down the list you can no longer rely on the seed, but have to list them manually in the exact order
they failed and tell pytest to not randomize them instead using `--random-order-bucket=none`, e.g.:

```bash
pytest --random-order-bucket=none tests/test_a.py tests/test_c.py tests/test_b.py
```

To disable the shuffling for all tests:

```bash
pytest --random-order-bucket=none
```

By default `--random-order-bucket=module` is implied, which will shuffle the files on the module levels. It can also
shuffle on `class`, `package`, `global` and `none` levels. For the complete details please see its
[documentation](https://github.com/jbasko/pytest-random-order).

Another randomization alternative is: [`pytest-randomly`](https://github.com/pytest-dev/pytest-randomly). This
module has a very similar functionality/interface, but it doesn't have the bucket modes available in
`pytest-random-order`. It has the same problem of imposing itself once installed.

### Look and feel variations

#### pytest-sugar

[pytest-sugar](https://github.com/Frozenball/pytest-sugar) is a plugin that improves the look-n-feel, adds a
progressbar, and show tests that fail and the assert instantly. It gets activated automatically upon installation.

```bash
pip install pytest-sugar
```

To run tests without it, run:

```bash
pytest -p no:sugar
```

or uninstall it.


key: _node_content
string_value:
{"id_": "b89107d2-4568-48fc-83db-1033f6272fda", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "10a2f61b-b843-40a6-896b-4436b347964c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "ef37c432059f55deb255561e23c016f4dfe697a451ad2c619e00127d6dae0b50", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0aab41fa-8484-41a5-be93-efa11215a205", "node_type": "1", "metadata": {}, "hash": "24f2914553df558d2923be8e790b04a4e9faf48c08ad450511382f210b06682f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11946, "end_char_idx": 14908, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Report each sub-test name and its progress

For a single or a group of tests via `pytest` (after `pip install pytest-pspec`):

```bash
pytest --pspec tests/test_optimization.py
```

#### Instantly shows failed tests

[pytest-instafail](https://github.com/pytest-dev/pytest-instafail) shows failures and errors instantly instead of
waiting until the end of test session.

```bash
pip install pytest-instafail
```

```bash
pytest --instafail
```

### To GPU or not to GPU

On a GPU-enabled setup, to test in CPU-only mode add `CUDA_VISIBLE_DEVICES=""`:

```bash
CUDA_VISIBLE_DEVICES="" pytest tests/utils/test_logging.py
```

or if you have multiple gpus, you can specify which one is to be used by `pytest`. For example, to use only the
second gpu if you have gpus `0` and `1`, you can run:

```bash
CUDA_VISIBLE_DEVICES="1" pytest tests/utils/test_logging.py
```

This is handy when you want to run different tasks on different GPUs.

Some tests must be run on CPU-only, others on either CPU or GPU or TPU, yet others on multiple-GPUs. The following skip
decorators are used to set the requirements of tests CPU/GPU/TPU-wise:

- `require_torch` - this test will run only under torch
- `require_torch_gpu` - as `require_torch` plus requires at least 1 GPU
- `require_torch_multi_gpu` - as `require_torch` plus requires at least 2 GPUs
- `require_torch_non_multi_gpu` - as `require_torch` plus requires 0 or 1 GPUs
- `require_torch_up_to_2_gpus` - as `require_torch` plus requires 0 or 1 or 2 GPUs
- `require_torch_xla` - as `require_torch` plus requires at least 1 TPU

Let's depict the GPU requirements in the following table:


| n gpus | decorator                      |
|--------|--------------------------------|
| `>= 0` | `@require_torch`               |
| `>= 1` | `@require_torch_gpu`           |
| `>= 2` | `@require_torch_multi_gpu`     |
| `< 2`  | `@require_torch_non_multi_gpu` |
| `< 3`  | `@require_torch_up_to_2_gpus`  |


For example, here is a test that must be run only when there are 2 or more GPUs available and pytorch is installed:

```python no-style
@require_torch_multi_gpu
def test_example_with_multi_gpu():
```

If a test requires `tensorflow` use the `require_tf` decorator. For example:

```python no-style
@require_tf
def test_tf_thing_with_tensorflow():
```

These decorators can be stacked. For example, if a test is slow and requires at least one GPU under pytorch, here is
how to set it up:

```python no-style
@require_torch_gpu
@slow
def test_example_slow_on_gpu():
```

Some decorators like `@parametrized` rewrite test names, therefore `@require_*` skip decorators have to be listed
last for them to work correctly. Here is an example of the correct usage:

```python no-style
@parameterized.expand(...)
@require_torch_multi_gpu
def test_integration_foo():
```

This order problem doesn't exist with `@pytest.mark.parametrize`, you can put it first or last and it will still
work. But it only works with non-unittests.


key: _node_content
string_value:
{"id_": "0aab41fa-8484-41a5-be93-efa11215a205", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b89107d2-4568-48fc-83db-1033f6272fda", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "7d1be0ff34044ea0ffde0eafe872af782131ee67d0ba09face6e98f0a5f1bf36", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2672bb5a-6dfe-43d5-8d46-9743fbf5fdb8", "node_type": "1", "metadata": {}, "hash": "5c029f3134ee9441daac81eecd7c004fddadc9dff8b502bd6f64118c78f104e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13580, "end_char_idx": 16612, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
| n gpus | decorator                      |
|--------|--------------------------------|
| `>= 0` | `@require_torch`               |
| `>= 1` | `@require_torch_gpu`           |
| `>= 2` | `@require_torch_multi_gpu`     |
| `< 2`  | `@require_torch_non_multi_gpu` |
| `< 3`  | `@require_torch_up_to_2_gpus`  |


For example, here is a test that must be run only when there are 2 or more GPUs available and pytorch is installed:

```python no-style
@require_torch_multi_gpu
def test_example_with_multi_gpu():
```

If a test requires `tensorflow` use the `require_tf` decorator. For example:

```python no-style
@require_tf
def test_tf_thing_with_tensorflow():
```

These decorators can be stacked. For example, if a test is slow and requires at least one GPU under pytorch, here is
how to set it up:

```python no-style
@require_torch_gpu
@slow
def test_example_slow_on_gpu():
```

Some decorators like `@parametrized` rewrite test names, therefore `@require_*` skip decorators have to be listed
last for them to work correctly. Here is an example of the correct usage:

```python no-style
@parameterized.expand(...)
@require_torch_multi_gpu
def test_integration_foo():
```

This order problem doesn't exist with `@pytest.mark.parametrize`, you can put it first or last and it will still
work. But it only works with non-unittests.

Inside tests:

- How many GPUs are available:

```python
from transformers.testing_utils import get_gpu_count

n_gpu = get_gpu_count()  # works with torch and tf
```

### Testing with a specific PyTorch backend or device

To run the test suite on a specific torch device add `TRANSFORMERS_TEST_DEVICE="$device"` where `$device` is the target backend. For example, to test on CPU only:

```bash
TRANSFORMERS_TEST_DEVICE="cpu" pytest tests/utils/test_logging.py
```

This variable is useful for testing custom or less common PyTorch backends such as `mps`, `xpu` or `npu`. It can also be used to achieve the same effect as `CUDA_VISIBLE_DEVICES` by targeting specific GPUs or testing in CPU-only mode.

Certain devices will require an additional import after importing `torch` for the first time. This can be specified using the environment variable `TRANSFORMERS_TEST_BACKEND`:

```bash
TRANSFORMERS_TEST_BACKEND="torch_npu" pytest tests/utils/test_logging.py
```
Alternative backends may also require the replacement of device-specific functions. For example `torch.cuda.manual_seed` may need to be replaced with a device-specific seed setter like `torch.npu.manual_seed` or `torch.xpu.manual_seed` to correctly set a random seed on the device. To specify a new backend with backend-specific device functions when running the test suite, create a Python device specification file `spec.py` in the format:

```python
import torch
import torch_npu # for xpu, replace it with `import intel_extension_for_pytorch`
# !! Further additional imports can be added here !!

# Specify the device name (eg. 'cuda', 'cpu', 'npu', 'xpu', 'mps')
DEVICE_NAME = 'npu'

# Specify device-specific backends to dispatch to.


key: _node_content
string_value:
{"id_": "2672bb5a-6dfe-43d5-8d46-9743fbf5fdb8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0aab41fa-8484-41a5-be93-efa11215a205", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "bfdc46dfe04527e6629d8fb806f49c2874b09733895810d90f8e27c0baf8542a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f6ab0c96-33e5-4639-93f7-267095db6a4f", "node_type": "1", "metadata": {}, "hash": "2a451f1fb0ffa1308309626b131e0faf8fed8ed244dea65a4d4670f8ca61da26", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14910, "end_char_idx": 18178, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Inside tests:

- How many GPUs are available:

```python
from transformers.testing_utils import get_gpu_count

n_gpu = get_gpu_count()  # works with torch and tf
```

### Testing with a specific PyTorch backend or device

To run the test suite on a specific torch device add `TRANSFORMERS_TEST_DEVICE="$device"` where `$device` is the target backend. For example, to test on CPU only:

```bash
TRANSFORMERS_TEST_DEVICE="cpu" pytest tests/utils/test_logging.py
```

This variable is useful for testing custom or less common PyTorch backends such as `mps`, `xpu` or `npu`. It can also be used to achieve the same effect as `CUDA_VISIBLE_DEVICES` by targeting specific GPUs or testing in CPU-only mode.

Certain devices will require an additional import after importing `torch` for the first time. This can be specified using the environment variable `TRANSFORMERS_TEST_BACKEND`:

```bash
TRANSFORMERS_TEST_BACKEND="torch_npu" pytest tests/utils/test_logging.py
```
Alternative backends may also require the replacement of device-specific functions. For example `torch.cuda.manual_seed` may need to be replaced with a device-specific seed setter like `torch.npu.manual_seed` or `torch.xpu.manual_seed` to correctly set a random seed on the device. To specify a new backend with backend-specific device functions when running the test suite, create a Python device specification file `spec.py` in the format:

```python
import torch
import torch_npu # for xpu, replace it with `import intel_extension_for_pytorch`
# !! Further additional imports can be added here !!

# Specify the device name (eg. 'cuda', 'cpu', 'npu', 'xpu', 'mps')
DEVICE_NAME = 'npu'

# Specify device-specific backends to dispatch to.
# If not specified, will fallback to 'default' in 'testing_utils.py`
MANUAL_SEED_FN = torch.npu.manual_seed
EMPTY_CACHE_FN = torch.npu.empty_cache
DEVICE_COUNT_FN = torch.npu.device_count
```
This format also allows for specification of any additional imports required. To use this file to replace equivalent methods in the test suite, set the environment variable `TRANSFORMERS_TEST_DEVICE_SPEC` to the path of the spec file, e.g. `TRANSFORMERS_TEST_DEVICE_SPEC=spec.py`.

Currently, only `MANUAL_SEED_FN`, `EMPTY_CACHE_FN` and `DEVICE_COUNT_FN` are supported for device-specific dispatch.

### Distributed training

`pytest` can't deal with distributed training directly. If this is attempted - the sub-processes don't do the right
thing and end up thinking they are `pytest` and start running the test suite in loops. It works, however, if one
spawns a normal process that then spawns off multiple workers and manages the IO pipes.

Here are some tests that use it:

- [test_trainer_distributed.py](https://github.com/huggingface/transformers/tree/main/tests/trainer/test_trainer_distributed.py)
- [test_deepspeed.py](https://github.com/huggingface/transformers/tree/main/tests/deepspeed/test_deepspeed.py)

To jump right into the execution point, search for the `execute_subprocess_async` call in those tests.

You will need at least 2 GPUs to see these tests in action:

```bash
CUDA_VISIBLE_DEVICES=0,1 RUN_SLOW=1 pytest -sv tests/test_trainer_distributed.py
```

### Output capture

During test execution any output sent to `stdout` and `stderr` is captured.


key: _node_content
string_value:
{"id_": "f6ab0c96-33e5-4639-93f7-267095db6a4f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2672bb5a-6dfe-43d5-8d46-9743fbf5fdb8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "66395a285565302c13bd0984807d199cb4aca13398ec0f645eae76e36204d33d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3cd48a76-ad09-4a8d-a678-d6064b66f041", "node_type": "1", "metadata": {}, "hash": "0af4d5c556c3125733e710f7efee3f543a79d1783a9ee4190ae684f181d7d7bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16613, "end_char_idx": 19893, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# If not specified, will fallback to 'default' in 'testing_utils.py`
MANUAL_SEED_FN = torch.npu.manual_seed
EMPTY_CACHE_FN = torch.npu.empty_cache
DEVICE_COUNT_FN = torch.npu.device_count
```
This format also allows for specification of any additional imports required. To use this file to replace equivalent methods in the test suite, set the environment variable `TRANSFORMERS_TEST_DEVICE_SPEC` to the path of the spec file, e.g. `TRANSFORMERS_TEST_DEVICE_SPEC=spec.py`.

Currently, only `MANUAL_SEED_FN`, `EMPTY_CACHE_FN` and `DEVICE_COUNT_FN` are supported for device-specific dispatch.

### Distributed training

`pytest` can't deal with distributed training directly. If this is attempted - the sub-processes don't do the right
thing and end up thinking they are `pytest` and start running the test suite in loops. It works, however, if one
spawns a normal process that then spawns off multiple workers and manages the IO pipes.

Here are some tests that use it:

- [test_trainer_distributed.py](https://github.com/huggingface/transformers/tree/main/tests/trainer/test_trainer_distributed.py)
- [test_deepspeed.py](https://github.com/huggingface/transformers/tree/main/tests/deepspeed/test_deepspeed.py)

To jump right into the execution point, search for the `execute_subprocess_async` call in those tests.

You will need at least 2 GPUs to see these tests in action:

```bash
CUDA_VISIBLE_DEVICES=0,1 RUN_SLOW=1 pytest -sv tests/test_trainer_distributed.py
```

### Output capture

During test execution any output sent to `stdout` and `stderr` is captured. If a test or a setup method fails, its
according captured output will usually be shown along with the failure traceback.

To disable output capturing and to get the `stdout` and `stderr` normally, use `-s` or `--capture=no`:

```bash
pytest -s tests/utils/test_logging.py
```

To send test results to JUnit format output:

```bash
pytest tests --junitxml=result.xml
```

### Color control

To have no color (e.g., yellow on white background is not readable):

```bash
pytest --color=no tests/utils/test_logging.py
```

### Sending test report to online pastebin service

Creating a URL for each test failure:

```bash
pytest --pastebin=failed tests/utils/test_logging.py
```

This will submit test run information to a remote Paste service and provide a URL for each failure. You may select
tests as usual or add for example -x if you only want to send one particular failure.

Creating a URL for a whole test session log:

```bash
pytest --pastebin=all tests/utils/test_logging.py
```

## Writing tests

🤗 transformers tests are based on `unittest`, but run by `pytest`, so most of the time features from both systems
can be used.

You can read [here](https://docs.pytest.org/en/stable/unittest.html) which features are supported, but the important
thing to remember is that most `pytest` fixtures don't work. Neither parametrization, but we use the module
`parameterized` that works in a similar way.


### Parametrization

Often, there is a need to run the same test multiple times, but with different arguments. It could be done from within
the test, but then there is no way of running that test for just one set of arguments.

```python
# test_this1.py
import unittest
from parameterized import parameterized


key: _node_content
string_value:
{"id_": "3cd48a76-ad09-4a8d-a678-d6064b66f041", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f6ab0c96-33e5-4639-93f7-267095db6a4f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "a7bf4583254cc529e653609f350ef06fcc2d27a1ee572b98952c6283b1a44d5c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "912d3c85-7542-41e0-bf0f-b6a36e9cfe4a", "node_type": "1", "metadata": {}, "hash": "d9b43fd7e002a4234d0ca023a2dbd2e534dfbe13b6e5b3af3fee55d81ab82fd6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18179, "end_char_idx": 21660, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If a test or a setup method fails, its
according captured output will usually be shown along with the failure traceback.

To disable output capturing and to get the `stdout` and `stderr` normally, use `-s` or `--capture=no`:

```bash
pytest -s tests/utils/test_logging.py
```

To send test results to JUnit format output:

```bash
pytest tests --junitxml=result.xml
```

### Color control

To have no color (e.g., yellow on white background is not readable):

```bash
pytest --color=no tests/utils/test_logging.py
```

### Sending test report to online pastebin service

Creating a URL for each test failure:

```bash
pytest --pastebin=failed tests/utils/test_logging.py
```

This will submit test run information to a remote Paste service and provide a URL for each failure. You may select
tests as usual or add for example -x if you only want to send one particular failure.

Creating a URL for a whole test session log:

```bash
pytest --pastebin=all tests/utils/test_logging.py
```

## Writing tests

🤗 transformers tests are based on `unittest`, but run by `pytest`, so most of the time features from both systems
can be used.

You can read [here](https://docs.pytest.org/en/stable/unittest.html) which features are supported, but the important
thing to remember is that most `pytest` fixtures don't work. Neither parametrization, but we use the module
`parameterized` that works in a similar way.


### Parametrization

Often, there is a need to run the same test multiple times, but with different arguments. It could be done from within
the test, but then there is no way of running that test for just one set of arguments.

```python
# test_this1.py
import unittest
from parameterized import parameterized


class TestMathUnitTest(unittest.TestCase):
    @parameterized.expand(
        [
            ("negative", -1.5, -2.0),
            ("integer", 1, 1.0),
            ("large fraction", 1.6, 1),
        ]
    )
    def test_floor(self, name, input, expected):
        assert_equal(math.floor(input), expected)
```

Now, by default this test will be run 3 times, each time with the last 3 arguments of `test_floor` being assigned the
corresponding arguments in the parameter list.

and you could run just the `negative` and `integer` sets of params with:

```bash
pytest -k "negative and integer" tests/test_mytest.py
```

or all but `negative` sub-tests, with:

```bash
pytest -k "not negative" tests/test_mytest.py
```

Besides using the `-k` filter that was just mentioned, you can find out the exact name of each sub-test and run any
or all of them using their exact names.

```bash
pytest test_this1.py --collect-only -q
```

and it will list:

```bash
test_this1.py::TestMathUnitTest::test_floor_0_negative
test_this1.py::TestMathUnitTest::test_floor_1_integer
test_this1.py::TestMathUnitTest::test_floor_2_large_fraction
```

So now you can run just 2 specific sub-tests:

```bash
pytest test_this1.py::TestMathUnitTest::test_floor_0_negative  test_this1.py::TestMathUnitTest::test_floor_1_integer
```

The module [parameterized](https://pypi.org/project/parameterized/) which is already in the developer dependencies
of `transformers` works for both: `unittests` and `pytest` tests.

If, however, the test is not a `unittest`, you may use `pytest.mark.parametrize` (or you may see it being used in
some existing tests, mostly under `examples`).

Here is the same example, this time using `pytest`'s `parametrize` marker:

```python
# test_this2.py
import pytest


key: _node_content
string_value:
{"id_": "912d3c85-7542-41e0-bf0f-b6a36e9cfe4a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3cd48a76-ad09-4a8d-a678-d6064b66f041", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "2794f8ffbdbb68be3fb14b5255e36317dc34240bdde5543c928168620047749e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e2fae82d-a4af-456d-bb34-655aa36ce3f6", "node_type": "1", "metadata": {}, "hash": "e1a9f85e7d761ccd28d2f659c8ec5ad1997f84394b3c97b38fde0f1b736b7a44", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21663, "end_char_idx": 24324, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
@pytest.mark.parametrize(
    "name, input, expected",
    [
        ("negative", -1.5, -2.0),
        ("integer", 1, 1.0),
        ("large fraction", 1.6, 1),
    ],
)
def test_floor(name, input, expected):
    assert_equal(math.floor(input), expected)
```

Same as with `parameterized`, with `pytest.mark.parametrize` you can have a fine control over which sub-tests are
run, if the `-k` filter doesn't do the job. Except, this parametrization function creates a slightly different set of
names for the sub-tests. Here is what they look like:

```bash
pytest test_this2.py --collect-only -q
```

and it will list:

```bash
test_this2.py::test_floor[integer-1-1.0]
test_this2.py::test_floor[negative--1.5--2.0]
test_this2.py::test_floor[large fraction-1.6-1]
```

So now you can run just the specific test:

```bash
pytest test_this2.py::test_floor[negative--1.5--2.0] test_this2.py::test_floor[integer-1-1.0]
```

as in the previous example.



### Files and directories

In tests often we need to know where things are relative to the current test file, and it's not trivial since the test
could be invoked from more than one directory or could reside in sub-directories with different depths. A helper class
`transformers.test_utils.TestCasePlus` solves this problem by sorting out all the basic paths and provides easy
accessors to them:

- `pathlib` objects (all fully resolved):

  - `test_file_path` - the current test file path, i.e. `__file__`
  - `test_file_dir` - the directory containing the current test file
  - `tests_dir` - the directory of the `tests` test suite
  - `examples_dir` - the directory of the `examples` test suite
  - `repo_root_dir` - the directory of the repository
  - `src_dir` - the directory of `src` (i.e. where the `transformers` sub-dir resides)

- stringified paths---same as above but these return paths as strings, rather than `pathlib` objects:

  - `test_file_path_str`
  - `test_file_dir_str`
  - `tests_dir_str`
  - `examples_dir_str`
  - `repo_root_dir_str`
  - `src_dir_str`

To start using those all you need is to make sure that the test resides in a subclass of
`transformers.test_utils.TestCasePlus`. For example:

```python
from transformers.testing_utils import TestCasePlus


class PathExampleTest(TestCasePlus):
    def test_something_involving_local_locations(self):
        data_dir = self.tests_dir / "fixtures/tests_samples/wmt_en_ro"
```

If you don't need to manipulate paths via `pathlib` or you just need a path as a string, you can always invoked
`str()` on the `pathlib` object or use the accessors ending with `_str`. For example:

```python
from transformers.testing_utils import TestCasePlus


key: _node_content
string_value:
{"id_": "e2fae82d-a4af-456d-bb34-655aa36ce3f6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "912d3c85-7542-41e0-bf0f-b6a36e9cfe4a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "5e050be3c4c20207eb7fb049d460761d86dae310f061471f59d46214c123bfaa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9bf95382-7516-487a-a906-6d688212efaf", "node_type": "1", "metadata": {}, "hash": "2fb931b43c9ad8db93408479f62947c456c4199844cd902abdc43be80ef1ecba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23895, "end_char_idx": 27221, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class PathExampleTest(TestCasePlus):
    def test_something_involving_local_locations(self):
        data_dir = self.tests_dir / "fixtures/tests_samples/wmt_en_ro"
```

If you don't need to manipulate paths via `pathlib` or you just need a path as a string, you can always invoked
`str()` on the `pathlib` object or use the accessors ending with `_str`. For example:

```python
from transformers.testing_utils import TestCasePlus


class PathExampleTest(TestCasePlus):
    def test_something_involving_stringified_locations(self):
        examples_dir = self.examples_dir_str
```

### Temporary files and directories

Using unique temporary files and directories are essential for parallel test running, so that the tests won't overwrite
each other's data. Also we want to get the temporary files and directories removed at the end of each test that created
them. Therefore, using packages like `tempfile`, which address these needs is essential.

However, when debugging tests, you need to be able to see what goes into the temporary file or directory and you want
to know it's exact path and not having it randomized on every test re-run.

A helper class `transformers.test_utils.TestCasePlus` is best used for such purposes. It's a sub-class of
`unittest.TestCase`, so we can easily inherit from it in the test modules.

Here is an example of its usage:

```python
from transformers.testing_utils import TestCasePlus


class ExamplesTests(TestCasePlus):
    def test_whatever(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
```

This code creates a unique temporary directory, and sets `tmp_dir` to its location.

- Create a unique temporary dir:

```python
def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir()
```

`tmp_dir` will contain the path to the created temporary dir. It will be automatically removed at the end of the
test.

- Create a temporary dir of my choice, ensure it's empty before the test starts and don't empty it after the test.

```python
def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir("./xxx")
```

This is useful for debug when you want to monitor a specific directory and want to make sure the previous tests didn't
leave any data in there.

- You can override the default behavior by directly overriding the `before` and `after` args, leading to one of the
  following behaviors:

  - `before=True`: the temporary dir will always be cleared at the beginning of the test.
  - `before=False`: if the temporary dir already existed, any existing files will remain there.
  - `after=True`: the temporary dir will always be deleted at the end of the test.
  - `after=False`: the temporary dir will always be left intact at the end of the test.

<Tip>

In order to run the equivalent of `rm -r` safely, only subdirs of the project repository checkout are allowed if
an explicit `tmp_dir` is used, so that by mistake no `/tmp` or similar important part of the filesystem will
get nuked. i.e. please always pass paths that start with `./`.

</Tip>

<Tip>

Each test can register multiple temporary directories and they all will get auto-removed, unless requested
otherwise.

</Tip>

### Temporary sys.path override

If you need to temporary override `sys.path` to import from another test for example, you can use the
`ExtendSysPath` context manager. Example:


key: _node_content
string_value:
{"id_": "9bf95382-7516-487a-a906-6d688212efaf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e2fae82d-a4af-456d-bb34-655aa36ce3f6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "7baf371e60302c790c5fe4978d6761f2af0d3032fecc11d6226d9ea5c600f815", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3eb49e8b-2fa0-4a8b-9f7b-fc2e9d3206cc", "node_type": "1", "metadata": {}, "hash": "74d4c3e2ea345f79bf0dbdf2c36584b8b9798f3ded4c59ca4c7ecd01471c3a58", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27224, "end_char_idx": 30254, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import os
from transformers.testing_utils import ExtendSysPath

bindir = os.path.abspath(os.path.dirname(__file__))
with ExtendSysPath(f"{bindir}/.."):
    from test_trainer import TrainerIntegrationCommon  # noqa
```

### Skipping tests

This is useful when a bug is found and a new test is written, yet the bug is not fixed yet. In order to be able to
commit it to the main repository we need make sure it's skipped during `make test`.

Methods:

-  A **skip** means that you expect your test to pass only if some conditions are met, otherwise pytest should skip
  running the test altogether. Common examples are skipping windows-only tests on non-windows platforms, or skipping
  tests that depend on an external resource which is not available at the moment (for example a database).

-  A **xfail** means that you expect a test to fail for some reason. A common example is a test for a feature not yet
  implemented, or a bug not yet fixed. When a test passes despite being expected to fail (marked with
  pytest.mark.xfail), it’s an xpass and will be reported in the test summary.

One of the important differences between the two is that `skip` doesn't run the test, and `xfail` does. So if the
code that's buggy causes some bad state that will affect other tests, do not use `xfail`.

#### Implementation

- Here is how to skip whole test unconditionally:

```python no-style
@unittest.skip(reason="this bug needs to be fixed")
def test_feature_x():
```

or via pytest:

```python no-style
@pytest.mark.skip(reason="this bug needs to be fixed")
```

or the `xfail` way:

```python no-style
@pytest.mark.xfail
def test_feature_x():
```


Here's how to skip a test based on internal checks within the test:

```python
def test_feature_x():
    if not has_something():
        pytest.skip("unsupported configuration")
```

or the whole module:

```python
import pytest

if not pytest.config.getoption("--custom-flag"):
    pytest.skip("--custom-flag is missing, skipping tests", allow_module_level=True)
```

or the `xfail` way:

```python
def test_feature_x():
    pytest.xfail("expected to fail until bug XYZ is fixed")
```

- Here is how to skip all tests in a module if some import is missing:

```python
docutils = pytest.importorskip("docutils", minversion="0.3")
```

-  Skip a test based on a condition:

```python no-style
@pytest.mark.skipif(sys.version_info < (3,6), reason="requires python3.6 or higher")
def test_feature_x():
```

or:

```python no-style
@unittest.skipIf(torch_device == "cpu", "Can't do half precision")
def test_feature_x():
```

or skip the whole module:

```python no-style
@pytest.mark.skipif(sys.platform == 'win32', reason="does not run on windows")
class TestClass():
    def test_feature_x(self):
```

More details, example and ways are [here](https://docs.pytest.org/en/latest/skipping.html).

### Slow tests

The library of tests is ever-growing, and some of the tests take minutes to run, therefore we can't afford waiting for
an hour for the test suite to complete on CI.


key: _node_content
string_value:
{"id_": "3eb49e8b-2fa0-4a8b-9f7b-fc2e9d3206cc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9bf95382-7516-487a-a906-6d688212efaf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08a4d27a29c44191303eec26f5bf1b071c9aec4994424b66eed4a5adc339f0ab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ad7f9a9-7732-4f7b-8ddb-edd1434647b8", "node_type": "1", "metadata": {}, "hash": "68842981d9d4c92acafa2028312c1a402e6a9f2898f5b65b69f4b1caaa5936d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28880, "end_char_idx": 32132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here's how to skip a test based on internal checks within the test:

```python
def test_feature_x():
    if not has_something():
        pytest.skip("unsupported configuration")
```

or the whole module:

```python
import pytest

if not pytest.config.getoption("--custom-flag"):
    pytest.skip("--custom-flag is missing, skipping tests", allow_module_level=True)
```

or the `xfail` way:

```python
def test_feature_x():
    pytest.xfail("expected to fail until bug XYZ is fixed")
```

- Here is how to skip all tests in a module if some import is missing:

```python
docutils = pytest.importorskip("docutils", minversion="0.3")
```

-  Skip a test based on a condition:

```python no-style
@pytest.mark.skipif(sys.version_info < (3,6), reason="requires python3.6 or higher")
def test_feature_x():
```

or:

```python no-style
@unittest.skipIf(torch_device == "cpu", "Can't do half precision")
def test_feature_x():
```

or skip the whole module:

```python no-style
@pytest.mark.skipif(sys.platform == 'win32', reason="does not run on windows")
class TestClass():
    def test_feature_x(self):
```

More details, example and ways are [here](https://docs.pytest.org/en/latest/skipping.html).

### Slow tests

The library of tests is ever-growing, and some of the tests take minutes to run, therefore we can't afford waiting for
an hour for the test suite to complete on CI. Therefore, with some exceptions for essential tests, slow tests should be
marked as in the example below:

```python no-style
from transformers.testing_utils import slow
@slow
def test_integration_foo():
```

Once a test is marked as `@slow`, to run such tests set `RUN_SLOW=1` env var, e.g.:

```bash
RUN_SLOW=1 pytest tests
```

Some decorators like `@parameterized` rewrite test names, therefore `@slow` and the rest of the skip decorators
`@require_*` have to be listed last for them to work correctly. Here is an example of the correct usage:

```python no-style
@parameterized.expand(...)
@slow
def test_integration_foo():
```

As explained at the beginning of this document, slow tests get to run on a scheduled basis, rather than in PRs CI
checks. So it's possible that some problems will be missed during a PR submission and get merged. Such problems will
get caught during the next scheduled CI job. But it also means that it's important to run the slow tests on your
machine before submitting the PR.

Here is a rough decision making mechanism for choosing which tests should be marked as slow:

If the test is focused on one of the library's internal components (e.g., modeling files, tokenization files,
pipelines), then we should run that test in the non-slow test suite. If it's focused on an other aspect of the library,
such as the documentation or the examples, then we should run these tests in the slow test suite. And then, to refine
this approach we should have exceptions:

- All tests that need to download a heavy set of weights or a dataset that is larger than ~50MB (e.g., model or
  tokenizer integration tests, pipeline integration tests) should be set to slow. If you're adding a new model, you
  should create and upload to the hub a tiny version of it (with random weights) for integration tests. This is
  discussed in the following paragraphs.


key: _node_content
string_value:
{"id_": "2ad7f9a9-7732-4f7b-8ddb-edd1434647b8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3eb49e8b-2fa0-4a8b-9f7b-fc2e9d3206cc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "a33cc0fa53f34a01935d2ba918fa973bb26afbcdb02be30571b1c8afbe502de6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a8a37756-f85a-4c9f-985f-4dd64e72d42a", "node_type": "1", "metadata": {}, "hash": "0007f7e2b27124ebd7f88da8ae267c5696e33e0ac98446b4dfa78b0544a036e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30546, "end_char_idx": 33817, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
:

```bash
RUN_SLOW=1 pytest tests
```

Some decorators like `@parameterized` rewrite test names, therefore `@slow` and the rest of the skip decorators
`@require_*` have to be listed last for them to work correctly. Here is an example of the correct usage:

```python no-style
@parameterized.expand(...)
@slow
def test_integration_foo():
```

As explained at the beginning of this document, slow tests get to run on a scheduled basis, rather than in PRs CI
checks. So it's possible that some problems will be missed during a PR submission and get merged. Such problems will
get caught during the next scheduled CI job. But it also means that it's important to run the slow tests on your
machine before submitting the PR.

Here is a rough decision making mechanism for choosing which tests should be marked as slow:

If the test is focused on one of the library's internal components (e.g., modeling files, tokenization files,
pipelines), then we should run that test in the non-slow test suite. If it's focused on an other aspect of the library,
such as the documentation or the examples, then we should run these tests in the slow test suite. And then, to refine
this approach we should have exceptions:

- All tests that need to download a heavy set of weights or a dataset that is larger than ~50MB (e.g., model or
  tokenizer integration tests, pipeline integration tests) should be set to slow. If you're adding a new model, you
  should create and upload to the hub a tiny version of it (with random weights) for integration tests. This is
  discussed in the following paragraphs.
- All tests that need to do a training not specifically optimized to be fast should be set to slow.
- We can introduce exceptions if some of these should-be-non-slow tests are excruciatingly slow, and set them to
  `@slow`. Auto-modeling tests, which save and load large files to disk, are a good example of tests that are marked
  as `@slow`.
- If a test completes under 1 second on CI (including downloads if any) then it should be a normal test regardless.

Collectively, all the non-slow tests need to cover entirely the different internals, while remaining fast. For example,
a significant coverage can be achieved by testing with specially created tiny models with random weights. Such models
have the very minimal number of layers (e.g., 2), vocab size (e.g., 1000), etc. Then the `@slow` tests can use large
slow models to do qualitative testing. To see the use of these simply look for *tiny* models with:

```bash
grep tiny tests examples
```

Here is an example of a [script](https://github.com/huggingface/transformers/tree/main/scripts/fsmt/fsmt-make-tiny-model.py) that created the tiny model
[stas/tiny-wmt19-en-de](https://huggingface.co/stas/tiny-wmt19-en-de). You can easily adjust it to your specific
model's architecture.

It's easy to measure the run-time incorrectly if for example there is an overheard of downloading a huge model, but if
you test it locally the downloaded files would be cached and thus the download time not measured. Hence check the
execution speed report in CI logs instead (the output of `pytest --durations=0 tests`).

That report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast.


key: _node_content
string_value:
{"id_": "a8a37756-f85a-4c9f-985f-4dd64e72d42a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ad7f9a9-7732-4f7b-8ddb-edd1434647b8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "6d6f8c2382c18b9ec73ed2e65d228f64360b76bbe22de693cfc5d450f83a5424", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "97388bc6-ffe1-4738-9473-048a91807205", "node_type": "1", "metadata": {}, "hash": "bb8fd2419b90aa9446a4b7ceda7da957800c8e340f9d0aa618d7dc27f5e51d88", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32233, "end_char_idx": 35277, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- We can introduce exceptions if some of these should-be-non-slow tests are excruciatingly slow, and set them to
  `@slow`. Auto-modeling tests, which save and load large files to disk, are a good example of tests that are marked
  as `@slow`.
- If a test completes under 1 second on CI (including downloads if any) then it should be a normal test regardless.

Collectively, all the non-slow tests need to cover entirely the different internals, while remaining fast. For example,
a significant coverage can be achieved by testing with specially created tiny models with random weights. Such models
have the very minimal number of layers (e.g., 2), vocab size (e.g., 1000), etc. Then the `@slow` tests can use large
slow models to do qualitative testing. To see the use of these simply look for *tiny* models with:

```bash
grep tiny tests examples
```

Here is an example of a [script](https://github.com/huggingface/transformers/tree/main/scripts/fsmt/fsmt-make-tiny-model.py) that created the tiny model
[stas/tiny-wmt19-en-de](https://huggingface.co/stas/tiny-wmt19-en-de). You can easily adjust it to your specific
model's architecture.

It's easy to measure the run-time incorrectly if for example there is an overheard of downloading a huge model, but if
you test it locally the downloaded files would be cached and thus the download time not measured. Hence check the
execution speed report in CI logs instead (the output of `pytest --durations=0 tests`).

That report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast.
If you notice that the test suite starts getting slow on CI, the top listing of this report will show the slowest
tests.


### Testing the stdout/stderr output

In order to test functions that write to `stdout` and/or `stderr`, the test can access those streams using the
`pytest`'s [capsys system](https://docs.pytest.org/en/latest/capture.html). Here is how this is accomplished:

```python
import sys


def print_to_stdout(s):
    print(s)


def print_to_stderr(s):
    sys.stderr.write(s)


def test_result_and_stdout(capsys):
    msg = "Hello"
    print_to_stdout(msg)
    print_to_stderr(msg)
    out, err = capsys.readouterr()  # consume the captured output streams
    # optional: if you want to replay the consumed streams:
    sys.stdout.write(out)
    sys.stderr.write(err)
    # test:
    assert msg in out
    assert msg in err
```

And, of course, most of the time, `stderr` will come as a part of an exception, so try/except has to be used in such
a case:

```python
def raise_exception(msg):
    raise ValueError(msg)


def test_something_exception():
    msg = "Not a good value"
    error = ""
    try:
        raise_exception(msg)
    except Exception as e:
        error = str(e)
        assert msg in error, f"{msg} is in the exception:\n{error}"
```

Another approach to capturing stdout is via `contextlib.redirect_stdout`:

```python
from io import StringIO
from contextlib import redirect_stdout


def print_to_stdout(s):
    print(s)


key: _node_content
string_value:
{"id_": "97388bc6-ffe1-4738-9473-048a91807205", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a8a37756-f85a-4c9f-985f-4dd64e72d42a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "adfa34a1a2f8bb19401cffdc122e4e020c75124cce3e9bb16110a8e6880f068b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c961aed9-3c6b-4d31-9106-bc9daf6d1cb5", "node_type": "1", "metadata": {}, "hash": "2edc364b2b73fe64690d6423c8613550bd19d2662997787614084d0db44a5db2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33698, "end_char_idx": 37171, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
That report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast.
If you notice that the test suite starts getting slow on CI, the top listing of this report will show the slowest
tests.


### Testing the stdout/stderr output

In order to test functions that write to `stdout` and/or `stderr`, the test can access those streams using the
`pytest`'s [capsys system](https://docs.pytest.org/en/latest/capture.html). Here is how this is accomplished:

```python
import sys


def print_to_stdout(s):
    print(s)


def print_to_stderr(s):
    sys.stderr.write(s)


def test_result_and_stdout(capsys):
    msg = "Hello"
    print_to_stdout(msg)
    print_to_stderr(msg)
    out, err = capsys.readouterr()  # consume the captured output streams
    # optional: if you want to replay the consumed streams:
    sys.stdout.write(out)
    sys.stderr.write(err)
    # test:
    assert msg in out
    assert msg in err
```

And, of course, most of the time, `stderr` will come as a part of an exception, so try/except has to be used in such
a case:

```python
def raise_exception(msg):
    raise ValueError(msg)


def test_something_exception():
    msg = "Not a good value"
    error = ""
    try:
        raise_exception(msg)
    except Exception as e:
        error = str(e)
        assert msg in error, f"{msg} is in the exception:\n{error}"
```

Another approach to capturing stdout is via `contextlib.redirect_stdout`:

```python
from io import StringIO
from contextlib import redirect_stdout


def print_to_stdout(s):
    print(s)


def test_result_and_stdout():
    msg = "Hello"
    buffer = StringIO()
    with redirect_stdout(buffer):
        print_to_stdout(msg)
    out = buffer.getvalue()
    # optional: if you want to replay the consumed streams:
    sys.stdout.write(out)
    # test:
    assert msg in out
```

An important potential issue with capturing stdout is that it may contain `\r` characters that in normal `print`
reset everything that has been printed so far. There is no problem with `pytest`, but with `pytest -s` these
characters get included in the buffer, so to be able to have the test run with and without `-s`, you have to make an
extra cleanup to the captured output, using `re.sub(r'~.*\r', '', buf, 0, re.M)`.

But, then we have a helper context manager wrapper to automatically take care of it all, regardless of whether it has
some `\r`'s in it or not, so it's a simple:

```python
from transformers.testing_utils import CaptureStdout

with CaptureStdout() as cs:
    function_that_writes_to_stdout()
print(cs.out)
```

Here is a full test example:

```python
from transformers.testing_utils import CaptureStdout

msg = "Secret message\r"
final = "Hello World"
with CaptureStdout() as cs:
    print(msg + final)
assert cs.out == final + "\n", f"captured: {cs.out}, expecting {final}"
```

If you'd like to capture `stderr` use the `CaptureStderr` class instead:

```python
from transformers.testing_utils import CaptureStderr

with CaptureStderr() as cs:
    function_that_writes_to_stderr()
print(cs.err)
```

If you need to capture both streams at once, use the parent `CaptureStd` class:

```python
from transformers.testing_utils import CaptureStd

with CaptureStd() as cs:
    function_that_writes_to_stdout_and_stderr()
print(cs.err, cs.out)
```

Also, to aid debugging test issues, by default these context managers automatically replay the captured streams on exit
from the context.


key: _node_content
string_value:
{"id_": "c961aed9-3c6b-4d31-9106-bc9daf6d1cb5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "97388bc6-ffe1-4738-9473-048a91807205", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "bf8300b0361ffa4ddf6166adb4d90bed9813297243de33216e1704e0bec3c0dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f2194be7-f4a0-448a-8d51-dfe1bf412443", "node_type": "1", "metadata": {}, "hash": "f45734368c77a8ef8f506540e9f9f591146d3397d76095912bb02705661f4375", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37174, "end_char_idx": 40305, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Capturing logger stream

If you need to validate the output of a logger, you can use `CaptureLogger`:

```python
from transformers import logging
from transformers.testing_utils import CaptureLogger

msg = "Testing 1, 2, 3"
logging.set_verbosity_info()
logger = logging.get_logger("transformers.models.bart.tokenization_bart")
with CaptureLogger(logger) as cl:
    logger.info(msg)
assert cl.out, msg + "\n"
```

### Testing with environment variables

If you want to test the impact of environment variables for a specific test you can use a helper decorator
`transformers.testing_utils.mockenv`

```python
from transformers.testing_utils import mockenv


class HfArgumentParserTest(unittest.TestCase):
    @mockenv(TRANSFORMERS_VERBOSITY="error")
    def test_env_override(self):
        env_level_str = os.getenv("TRANSFORMERS_VERBOSITY", None)
```

At times an external program needs to be called, which requires setting `PYTHONPATH` in `os.environ` to include
multiple local paths. A helper class `transformers.test_utils.TestCasePlus` comes to help:

```python
from transformers.testing_utils import TestCasePlus


class EnvExampleTest(TestCasePlus):
    def test_external_prog(self):
        env = self.get_env()
        # now call the external program, passing `env` to it
```

Depending on whether the test file was under the `tests` test suite or `examples` it'll correctly set up
`env[PYTHONPATH]` to include one of these two directories, and also the `src` directory to ensure the testing is
done against the current repo, and finally with whatever `env[PYTHONPATH]` was already set to before the test was
called if anything.

This helper method creates a copy of the `os.environ` object, so the original remains intact.


### Getting reproducible results

In some situations you may want to remove randomness for your tests. To get identical reproducible results set, you
will need to fix the seed:

```python
seed = 42

# python RNG
import random

random.seed(seed)

# pytorch RNGs
import torch

torch.manual_seed(seed)
torch.backends.cudnn.deterministic = True
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)

# numpy RNG
import numpy as np

np.random.seed(seed)

# tf RNG
tf.random.set_seed(seed)
```

### Debugging tests

To start a debugger at the point of the warning, do this:

```bash
pytest tests/utils/test_logging.py -W error::UserWarning --pdb
```

## Working with github actions workflows

To trigger a self-push workflow CI job, you must:

1. Create a new branch on `transformers` origin (not a fork!).
2. The branch name has to start with either `ci_` or `ci-` (`main` triggers it too, but we can't do PRs on
   `main`). It also gets triggered only for specific paths - you can find the up-to-date definition in case it
   changed since this document has been written [here](https://github.com/huggingface/transformers/blob/main/.github/workflows/self-push.yml) under *push:*
3. Create a PR from this branch.
4. Then you can see the job appear [here](https://github.com/huggingface/transformers/actions/workflows/self-push.yml). It may not run right away if there
   is a backlog.


key: _node_content
string_value:
{"id_": "f2194be7-f4a0-448a-8d51-dfe1bf412443", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9560cb24-a7be-444a-94da-06e15088e207", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "08dc83b60c4104dc38de2c5e49b9ddcf594caf4ed240524b8ab46f2e9b456bb0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c961aed9-3c6b-4d31-9106-bc9daf6d1cb5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/testing/", "title": "Testing", "tokens": 10558, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "1aa5e299ed384ab3f521bbceda70379275e3922d77d26da67f22af3cbdbaf72c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38913, "end_char_idx": 43501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


